2024-02-05T04:20:34.381 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-02-05T04:20:34.381 DEBUG:teuthology.run:Teuthology command: teuthology --name pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi --owner scheduled_pdiazbou@teuthology --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} --verbose --archive /home/teuthworker/archive/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404 -- /home/teuthworker/archive/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404/orig.config.yaml 2024-02-05T04:20:34.407 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T04:20:34.481 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404 branch: wip-pereman2-buffer-space description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: pdiazbou@redhat.com first_in_suite: false job_id: '7536404' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pereman2-buffer-space ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 workunit: branch: wip-pereman2-buffer-space sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 owner: scheduled_pdiazbou@teuthology priority: 200 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2470 sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 sleep_before_teardown: 0 subset: 512/20000 suite: rados suite_branch: wip-pereman2-buffer-space suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 targets: smithi188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDorhZVwQxTa4xYl3O/TUIrw9WBMlK2qSvz5WHTkS8iDHdf2Iips913ViLWYawfBhXCurZRqRwd8qNWadnAlVaM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-29_15:40:07 tube: smithi user: pdiazbou verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-05T04:20:34.481 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa; will attempt to use it 2024-02-05T04:20:34.481 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks 2024-02-05T04:20:34.482 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-05T04:20:34.483 INFO:teuthology.task.internal:Checking packages... 2024-02-05T04:20:34.510 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7' 2024-02-05T04:20:34.510 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-05T04:20:34.510 INFO:teuthology.packaging:ref: None 2024-02-05T04:20:34.511 INFO:teuthology.packaging:tag: None 2024-02-05T04:20:34.511 INFO:teuthology.packaging:branch: wip-pereman2-buffer-space 2024-02-05T04:20:34.511 INFO:teuthology.packaging:sha1: 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-05T04:20:34.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pereman2-buffer-space 2024-02-05T04:20:34.897 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-432.g3cfd0d29 2024-02-05T04:20:34.899 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-05T04:20:34.904 INFO:teuthology.task.internal:no buildpackages task found 2024-02-05T04:20:34.904 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-05T04:20:34.939 INFO:teuthology.task.internal:Saving configuration 2024-02-05T04:20:34.949 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-05T04:20:34.965 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T04:20:34.986 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:15:29.696964', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDorhZVwQxTa4xYl3O/TUIrw9WBMlK2qSvz5WHTkS8iDHdf2Iips913ViLWYawfBhXCurZRqRwd8qNWadnAlVaM='} 2024-02-05T04:20:34.986 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-05T04:20:34.992 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-02-05T04:20:34.992 INFO:teuthology.run_tasks:Running task console_log... 2024-02-05T04:20:35.056 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f574bcf7160>, signals=[15]) 2024-02-05T04:20:35.056 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-05T04:20:35.062 INFO:teuthology.task.internal:Opening connections... 2024-02-05T04:20:35.062 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2024-02-05T04:20:35.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:20:35.153 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-05T04:20:35.160 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2024-02-05T04:20:35.220 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2024-02-05T04:20:35.221 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2024-02-05T04:20:35.277 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="9" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="9" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el9" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:LOGO="fedora-logo-icon" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2024-02-05T04:20:35.278 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T04:20:35.279 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T04:20:35.279 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T04:20:35.279 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2024-02-05T04:20:35.304 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-05T04:20:35.312 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-05T04:20:35.347 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-05T04:20:35.348 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2024-02-05T04:20:35.363 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-05T04:20:35.380 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-05T04:20:35.381 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2024-02-05T04:20:35.499 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-05T04:20:35.531 INFO:teuthology.run_tasks:Running task kernel... 2024-02-05T04:20:35.546 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-05T04:20:35.547 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-05T04:20:35.547 DEBUG:teuthology.orchestra.run.smithi188:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T04:20:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:20:35.563 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-02-05T04:20:35.624 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-361.el9.x86_64 2024-02-05T04:20:35.624 INFO:teuthology.task.kernel:Running kernel on smithi188: 5.14.0-361.el9.x86_64 2024-02-05T04:20:35.624 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2024-02-05T04:20:38.070 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - AppStream 8.3 MB/s | 6.6 MB 00:00 2024-02-05T04:20:39.703 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-02-05T04:20:42.652 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-02-05T04:20:46.847 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-02-05T04:20:57.943 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 800 kB/s | 5.9 MB 00:07 2024-02-05T04:21:00.801 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 8.5 kB/s | 15 kB 00:01 2024-02-05T04:21:39.212 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 544 kB/s | 20 MB 00:38 2024-02-05T04:21:44.061 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-02-05T04:21:46.268 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T04:21:46.269 INFO:teuthology.orchestra.run.smithi188.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T04:21:46.354 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T04:21:46.365 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout:Install 4 Packages 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 99 M 2024-02-05T04:21:46.366 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 122 M 2024-02-05T04:21:46.367 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:21:47.221 INFO:teuthology.orchestra.run.smithi188.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 7.0 MB/s | 6.0 MB 00:00 2024-02-05T04:21:48.723 INFO:teuthology.orchestra.run.smithi188.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 8.8 MB/s | 21 MB 00:02 2024-02-05T04:21:49.079 INFO:teuthology.orchestra.run.smithi188.stdout:(3/4): kernel-modules-core-5.14.0-416.el9.x86_6 18 MB/s | 33 MB 00:01 2024-02-05T04:21:50.096 INFO:teuthology.orchestra.run.smithi188.stdout:(4/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 10 MB/s | 39 MB 00:03 2024-02-05T04:21:50.099 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:21:50.099 INFO:teuthology.orchestra.run.smithi188.stdout:Total 27 MB/s | 99 MB 00:03 2024-02-05T04:21:50.100 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:21:50.518 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:21:50.518 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:21:51.588 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:21:51.588 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:21:53.984 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:21:54.653 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:21:54.797 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:21:57.535 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:21:57.718 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:22:02.410 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:22:02.597 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:22:07.363 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:22:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:22:36.469 INFO:teuthology.orchestra.run.smithi188.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T04:22:36.469 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:22:36.538 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:22:36.968 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:22:36.968 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T04:22:36.968 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T04:22:36.968 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T04:22:37.407 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T04:22:37.407 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:22:37.407 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:22:37.407 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T04:22:37.407 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T04:22:37.407 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T04:22:37.407 INFO:teuthology.orchestra.run.smithi188.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T04:22:37.408 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:22:37.408 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:22:37.928 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:22:38.059 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:22:38.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:22:38.060 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T04:22:38.060 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T04:22:38.060 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T04:22:38.060 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-05T04:22:38.060 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi188.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T04:22:38.060 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T04:22:38.168 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T04:22:38.169 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T04:22:38.169 DEBUG:teuthology.orchestra.run.smithi188:> sudo rpm -qi grub2-tools 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Name : grub2-tools 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Epoch : 1 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Version : 2.06 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Release : 68.el9 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Architecture: x86_64 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Group : Unspecified 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Size : 8264254 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:License : GPLv3+ 2024-02-05T04:22:38.213 INFO:teuthology.orchestra.run.smithi188.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:Packager : builder@centos.org 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:Vendor : CentOS 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:Summary : Support tools for GRUB. 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:Description : 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:hardware devices. 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:22:38.214 INFO:teuthology.orchestra.run.smithi188.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T04:22:38.215 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T04:22:38.215 INFO:teuthology.task.kernel:Updating grub on smithi188 to boot 5.14.0-416.el9.x86_64 2024-02-05T04:22:38.216 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T04:22:38.782 INFO:teuthology.orchestra.run.smithi188.stderr:Generating grub configuration file ... 2024-02-05T04:22:40.110 INFO:teuthology.orchestra.run.smithi188.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T04:22:40.139 INFO:teuthology.orchestra.run.smithi188.stderr:done 2024-02-05T04:22:40.141 DEBUG:teuthology.orchestra.run.smithi188:> mktemp 2024-02-05T04:22:40.156 INFO:teuthology.orchestra.run.smithi188.stdout:/tmp/tmp.K0WIIBR4X7 2024-02-05T04:22:40.157 DEBUG:teuthology.orchestra.run.smithi188:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.K0WIIBR4X7 2024-02-05T04:22:40.227 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod 0666 /tmp/tmp.K0WIIBR4X7 2024-02-05T04:22:40.363 DEBUG:teuthology.orchestra.remote:smithi188:/tmp/tmp.K0WIIBR4X7 is 6KB 2024-02-05T04:22:40.411 DEBUG:teuthology.orchestra.run.smithi188:> rm -fr /tmp/tmp.K0WIIBR4X7 2024-02-05T04:22:40.425 DEBUG:teuthology.orchestra.run.smithi188:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T04:22:40.494 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T04:22:40.494 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T04:22:40.494 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T04:22:40.494 INFO:teuthology.orchestra.run.smithi188.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T04:22:40.496 DEBUG:teuthology.orchestra.run.smithi188:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T04:22:40.630 DEBUG:teuthology.orchestra.run.smithi188:> sudo shutdown -r now 2024-02-05T04:23:10.656 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-05T04:23:10.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-02-05T04:23:10.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:23:29.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-02-05T04:23:38.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-02-05T04:23:38.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:23:41.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-02-05T04:23:53.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-02-05T04:23:53.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:24:11.580 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2024-02-05T04:24:26.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-02-05T04:24:26.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:24:26.983 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-02-05T04:24:27.548 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-02-05T04:24:27.548 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-416.el9.x86_64"... 2024-02-05T04:24:27.548 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2024-02-05T04:24:27.576 INFO:teuthology.orchestra.run.smithi188.stdout:5.14.0-416.el9.x86_64 2024-02-05T04:24:27.576 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T04:24:27.576 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T04:24:27.576 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T04:24:28.577 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-05T04:24:28.577 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T04:24:28.745 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2024-02-05T04:24:28.867 DEBUG:teuthology.parallel:result is None 2024-02-05T04:24:28.867 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-05T04:24:28.875 INFO:teuthology.task.internal:Creating test directory... 2024-02-05T04:24:28.875 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T04:24:28.890 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-05T04:24:28.923 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-05T04:24:28.972 INFO:teuthology.task.internal:Creating archive directory... 2024-02-05T04:24:28.972 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T04:24:29.006 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-05T04:24:29.022 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-05T04:24:29.023 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T04:24:29.081 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:24:29.094 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:24:29.096 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-05T04:24:29.103 INFO:teuthology.task.internal:Configuring sudo... 2024-02-05T04:24:29.103 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T04:24:29.166 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-05T04:24:29.175 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-05T04:24:29.175 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T04:24:29.225 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:24:29.309 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T04:24:29.369 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:24:29.453 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T04:24:29.500 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-02-05T04:24:29.500 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T04:24:29.571 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2024-02-05T04:24:29.663 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T04:24:30.176 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-05T04:24:30.184 INFO:teuthology.task.internal:Starting timer... 2024-02-05T04:24:30.184 INFO:teuthology.run_tasks:Running task pcp... 2024-02-05T04:24:30.234 INFO:teuthology.run_tasks:Running task selinux... 2024-02-05T04:24:30.283 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2024-02-05T04:24:30.347 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: 2024-02-05T04:24:30.349 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-05T04:24:30.349 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2024-02-05T04:24:30.369 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2024-02-05T04:24:30.370 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2024-02-05T04:24:30.370 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T04:24:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:24:30.437 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-05T04:24:30.438 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2024-02-05T04:24:30.512 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-05T04:24:30.522 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-05T04:24:30.538 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-05T04:24:30.539 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-05T04:28:35.391 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi188.front.sepia.ceph.com')] 2024-02-05T04:28:35.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-02-05T04:28:35.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T04:28:35.486 DEBUG:teuthology.orchestra.run.smithi188:> true 2024-02-05T04:28:35.562 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi188.front.sepia.ceph.com' 2024-02-05T04:28:35.562 INFO:teuthology.run_tasks:Running task clock... 2024-02-05T04:28:35.574 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-05T04:28:35.574 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T04:28:35.575 DEBUG:teuthology.orchestra.run.smithi188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T04:28:35.633 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T04:28:35.649 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T04:28:35.678 INFO:teuthology.orchestra.run.smithi188.stderr:sudo: ntpd: command not found 2024-02-05T04:28:35.692 INFO:teuthology.orchestra.run.smithi188.stdout:506 Cannot talk to daemon 2024-02-05T04:28:35.708 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T04:28:35.724 INFO:teuthology.orchestra.run.smithi188.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T04:28:35.775 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-02-05T04:28:35.779 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T04:28:35.779 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-02-05T04:28:35.779 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:28:35.779 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:28:35.779 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:28:35.779 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T04:28:35.780 INFO:teuthology.run_tasks:Running task install... 2024-02-05T04:28:35.788 DEBUG:teuthology.task.install:project ceph 2024-02-05T04:28:35.788 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'}} 2024-02-05T04:28:35.788 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'} 2024-02-05T04:28:35.788 INFO:teuthology.task.install:Using flavor: default 2024-02-05T04:28:35.794 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-05T04:28:35.794 INFO:teuthology.task.install:extra packages: [] 2024-02-05T04:28:35.795 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7', 'tag': None, 'wait_for_package': False} 2024-02-05T04:28:35.795 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-05T04:28:36.181 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/centos/9/flavors/default/ 2024-02-05T04:28:36.181 INFO:teuthology.task.install.rpm:Package version is 19.0.0-432.g3cfd0d29 2024-02-05T04:28:36.300 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pereman2-buffer-space/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T04:28:36.300 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-02-05T04:28:36.300 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T04:28:36.329 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T04:28:36.330 DEBUG:teuthology.orchestra.run.smithi188:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T04:28:36.399 DEBUG:teuthology.orchestra.run.smithi188:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T04:28:36.480 DEBUG:teuthology.orchestra.run.smithi188:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T04:28:36.547 INFO:teuthology.orchestra.run.smithi188.stdout:check_obsoletes = 1 2024-02-05T04:28:36.549 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum clean all 2024-02-05T04:28:36.800 INFO:teuthology.orchestra.run.smithi188.stdout:74 files removed 2024-02-05T04:28:36.829 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-radosgw 2024-02-05T04:28:37.666 INFO:teuthology.orchestra.run.smithi188.stdout:ceph packages for x86_64 154 kB/s | 80 kB 00:00 2024-02-05T04:28:38.183 INFO:teuthology.orchestra.run.smithi188.stdout:ceph noarch packages 20 kB/s | 9.5 kB 00:00 2024-02-05T04:28:38.625 INFO:teuthology.orchestra.run.smithi188.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-02-05T04:28:39.293 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - AppStream 10 MB/s | 6.6 MB 00:00 2024-02-05T04:28:40.936 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-02-05T04:28:42.468 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-02-05T04:28:44.371 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-02-05T04:28:50.541 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 5.9 MB 00:02 2024-02-05T04:28:52.081 INFO:teuthology.orchestra.run.smithi188.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2024-02-05T04:29:46.906 INFO:teuthology.orchestra.run.smithi188.stdout:Extra Packages for Enterprise Linux 380 kB/s | 20 MB 00:54 2024-02-05T04:29:51.729 INFO:teuthology.orchestra.run.smithi188.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-02-05T04:29:54.316 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:29:54.328 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:29:54.328 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:29:54.328 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:29:54.328 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:29:54.328 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 8.6 M 2024-02-05T04:29:54.328 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrading: 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: librados2 x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 3.4 M 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1 x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 3.1 M 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 5.1 M 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 19 M 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 25 k 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2 x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 742 k 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T04:29:54.329 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1 x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 498 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2 x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 4.8 M 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 45 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 130 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 162 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 320 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 298 k 2024-02-05T04:29:54.330 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 99 k 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout:Install 27 Packages 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout:Upgrade 2 Packages 2024-02-05T04:29:54.331 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:29:54.332 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 55 M 2024-02-05T04:29:54.332 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:29:56.197 INFO:teuthology.orchestra.run.smithi188.stdout:(1/29): ceph-base-19.0.0-432.g3cfd0d29.el9.x86_ 2.9 MB/s | 5.1 MB 00:01 2024-02-05T04:29:56.314 INFO:teuthology.orchestra.run.smithi188.stdout:(2/29): ceph-selinux-19.0.0-432.g3cfd0d29.el9.x 216 kB/s | 25 kB 00:00 2024-02-05T04:29:56.489 INFO:teuthology.orchestra.run.smithi188.stdout:(3/29): libcephfs2-19.0.0-432.g3cfd0d29.el9.x86 4.2 MB/s | 742 kB 00:00 2024-02-05T04:29:56.623 INFO:teuthology.orchestra.run.smithi188.stdout:(4/29): libradosstriper1-19.0.0-432.g3cfd0d29.e 3.6 MB/s | 498 kB 00:00 2024-02-05T04:29:56.989 INFO:teuthology.orchestra.run.smithi188.stdout:(5/29): ceph-common-19.0.0-432.g3cfd0d29.el9.x8 7.5 MB/s | 19 MB 00:02 2024-02-05T04:29:57.106 INFO:teuthology.orchestra.run.smithi188.stdout:(6/29): python3-ceph-argparse-19.0.0-432.g3cfd0 388 kB/s | 45 kB 00:00 2024-02-05T04:29:57.231 INFO:teuthology.orchestra.run.smithi188.stdout:(7/29): python3-ceph-common-19.0.0-432.g3cfd0d2 1.0 MB/s | 130 kB 00:00 2024-02-05T04:29:57.382 INFO:teuthology.orchestra.run.smithi188.stdout:(8/29): librgw2-19.0.0-432.g3cfd0d29.el9.x86_64 6.3 MB/s | 4.8 MB 00:00 2024-02-05T04:29:57.407 INFO:teuthology.orchestra.run.smithi188.stdout:(9/29): python3-cephfs-19.0.0-432.g3cfd0d29.el9 923 kB/s | 162 kB 00:00 2024-02-05T04:29:57.482 INFO:teuthology.orchestra.run.smithi188.stdout:(10/29): python3-rados-19.0.0-432.g3cfd0d29.el9 3.1 MB/s | 320 kB 00:00 2024-02-05T04:29:57.508 INFO:teuthology.orchestra.run.smithi188.stdout:(11/29): python3-rbd-19.0.0-432.g3cfd0d29.el9.x 2.9 MB/s | 298 kB 00:00 2024-02-05T04:29:57.591 INFO:teuthology.orchestra.run.smithi188.stdout:(12/29): python3-rgw-19.0.0-432.g3cfd0d29.el9.x 916 kB/s | 99 kB 00:00 2024-02-05T04:29:57.758 INFO:teuthology.orchestra.run.smithi188.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 423 kB/s | 106 kB 00:00 2024-02-05T04:29:57.833 INFO:teuthology.orchestra.run.smithi188.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 590 kB/s | 44 kB 00:00 2024-02-05T04:29:57.859 INFO:teuthology.orchestra.run.smithi188.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 596 kB/s | 159 kB 00:00 2024-02-05T04:29:58.034 INFO:teuthology.orchestra.run.smithi188.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 661 kB 00:00 2024-02-05T04:29:58.059 INFO:teuthology.orchestra.run.smithi188.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-02-05T04:29:58.101 INFO:teuthology.orchestra.run.smithi188.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 620 kB/s | 41 kB 00:00 2024-02-05T04:29:58.552 INFO:teuthology.orchestra.run.smithi188.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 628 kB/s | 309 kB 00:00 2024-02-05T04:29:58.618 INFO:teuthology.orchestra.run.smithi188.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-02-05T04:29:58.677 INFO:teuthology.orchestra.run.smithi188.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 837 kB/s | 49 kB 00:00 2024-02-05T04:29:58.794 INFO:teuthology.orchestra.run.smithi188.stdout:(22/29): ceph-radosgw-19.0.0-432.g3cfd0d29.el9. 2.0 MB/s | 8.6 MB 00:04 2024-02-05T04:29:58.819 INFO:teuthology.orchestra.run.smithi188.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 474 kB/s | 67 kB 00:00 2024-02-05T04:29:59.127 INFO:teuthology.orchestra.run.smithi188.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 621 kB/s | 192 kB 00:00 2024-02-05T04:29:59.253 INFO:teuthology.orchestra.run.smithi188.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-02-05T04:29:59.711 INFO:teuthology.orchestra.run.smithi188.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-02-05T04:30:00.703 INFO:teuthology.orchestra.run.smithi188.stdout:(27/29): librados2-19.0.0-432.g3cfd0d29.el9.x86 2.3 MB/s | 3.4 MB 00:01 2024-02-05T04:30:00.803 INFO:teuthology.orchestra.run.smithi188.stdout:(28/29): librbd1-19.0.0-432.g3cfd0d29.el9.x86_6 2.9 MB/s | 3.1 MB 00:01 2024-02-05T04:30:02.254 INFO:teuthology.orchestra.run.smithi188.stdout:(29/29): libarrow-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:04 2024-02-05T04:30:02.260 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:30:02.261 INFO:teuthology.orchestra.run.smithi188.stdout:Total 7.0 MB/s | 55 MB 00:07 2024-02-05T04:30:02.261 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:30:02.297 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:30:02.297 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:30:02.754 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:30:02.755 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:30:04.164 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:30:04.256 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T04:30:04.291 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T04:30:04.342 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T04:30:04.671 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T04:30:04.723 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librados2-2:19.0.0-432.g3cfd0d29.el9.x86_64 5/31 2024-02-05T04:30:04.800 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:19.0.0-432.g3cfd0d29.el9.x86_64 5/31 2024-02-05T04:30:04.885 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rados-2:19.0.0-432.g3cfd0d29.el9.x86_64 6/31 2024-02-05T04:30:04.916 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs2-2:19.0.0-432.g3cfd0d29.el9.x86_64 7/31 2024-02-05T04:30:04.961 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs2-2:19.0.0-432.g3cfd0d29.el9.x86_64 7/31 2024-02-05T04:30:04.991 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T04:30:05.037 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-argparse-2:19.0.0-432.g3cfd0d29.el9.x 9/31 2024-02-05T04:30:05.102 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cephfs-2:19.0.0-432.g3cfd0d29.el9.x86_64 10/31 2024-02-05T04:30:05.127 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libradosstriper1-2:19.0.0-432.g3cfd0d29.el9.x86_64 11/31 2024-02-05T04:30:05.192 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libradosstriper1-2:19.0.0-432.g3cfd0d29.el9.x86_64 11/31 2024-02-05T04:30:05.300 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T04:30:05.335 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T04:30:05.375 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T04:30:05.416 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T04:30:05.444 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T04:30:05.477 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T04:30:05.787 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T04:30:05.815 INFO:teuthology.orchestra.run.smithi188.stdout: Upgrading : librbd1-2:19.0.0-432.g3cfd0d29.el9.x86_64 19/31 2024-02-05T04:30:05.888 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:19.0.0-432.g3cfd0d29.el9.x86_64 19/31 2024-02-05T04:30:05.920 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 20/31 2024-02-05T04:30:05.955 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T04:30:06.387 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T04:30:06.418 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librgw2-2:19.0.0-432.g3cfd0d29.el9.x86_64 23/31 2024-02-05T04:30:06.488 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librgw2-2:19.0.0-432.g3cfd0d29.el9.x86_64 23/31 2024-02-05T04:30:06.556 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 24/31 2024-02-05T04:30:06.587 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86 25/31 2024-02-05T04:30:08.733 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86_64 26/31 2024-02-05T04:30:08.783 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86_64 26/31 2024-02-05T04:30:09.327 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86_64 26/31 2024-02-05T04:30:09.370 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-base-2:19.0.0-432.g3cfd0d29.el9.x86_64 27/31 2024-02-05T04:30:09.461 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-base-2:19.0.0-432.g3cfd0d29.el9.x86_64 27/31 2024-02-05T04:30:09.461 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T04:30:09.461 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T04:30:09.462 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:09.504 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-selinux-2:19.0.0-432.g3cfd0d29.el9.x86_64 28/31 2024-02-05T04:30:19.015 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-selinux-2:19.0.0-432.g3cfd0d29.el9.x86_64 28/31 2024-02-05T04:30:19.015 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /sys 2024-02-05T04:30:19.015 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /proc 2024-02-05T04:30:19.015 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /mnt 2024-02-05T04:30:19.015 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /var/tmp 2024-02-05T04:30:19.016 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /home 2024-02-05T04:30:19.016 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /root 2024-02-05T04:30:19.016 INFO:teuthology.orchestra.run.smithi188.stdout:skipping the directory /tmp 2024-02-05T04:30:19.016 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:19.844 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-radosgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 29/31 2024-02-05T04:30:19.868 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 29/31 2024-02-05T04:30:19.868 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:30:19.869 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T04:30:19.869 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:30:19.869 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T04:30:19.869 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:19.870 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:30:19.963 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T04:30:19.963 INFO:teuthology.orchestra.run.smithi188.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-base-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86_64 2/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-radosgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 3/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-selinux-2:19.0.0-432.g3cfd0d29.el9.x86_64 4/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs2-2:19.0.0-432.g3cfd0d29.el9.x86_64 5/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libradosstriper1-2:19.0.0-432.g3cfd0d29.el9.x86_64 6/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librgw2-2:19.0.0-432.g3cfd0d29.el9.x86_64 7/31 2024-02-05T04:30:21.113 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-argparse-2:19.0.0-432.g3cfd0d29.el9.x 8/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86 9/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cephfs-2:19.0.0-432.g3cfd0d29.el9.x86_64 10/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rados-2:19.0.0-432.g3cfd0d29.el9.x86_64 11/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 12/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 13/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T04:30:21.114 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:19.0.0-432.g3cfd0d29.el9.x86_64 28/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T04:30:21.115 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:19.0.0-432.g3cfd0d29.el9.x86_64 30/31 2024-02-05T04:30:22.013 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T04:30:22.013 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:22.013 INFO:teuthology.orchestra.run.smithi188.stdout:Upgraded: 2024-02-05T04:30:22.013 INFO:teuthology.orchestra.run.smithi188.stdout: librados2-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.013 INFO:teuthology.orchestra.run.smithi188.stdout: librbd1-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.013 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-base-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-radosgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-selinux-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs2-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T04:30:22.014 INFO:teuthology.orchestra.run.smithi188.stdout: libradosstriper1-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: librgw2-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-argparse-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ceph-common-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cephfs-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rados-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:22.015 INFO:teuthology.orchestra.run.smithi188.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T04:30:22.016 INFO:teuthology.orchestra.run.smithi188.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T04:30:22.016 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:22.016 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:30:22.388 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-test 2024-02-05T04:30:22.966 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:31 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:30:23.122 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 45 M 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 163 k 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T04:30:23.123 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:23.124 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:30:23.124 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:23.124 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2024-02-05T04:30:23.124 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:23.124 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 46 M 2024-02-05T04:30:23.124 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 196 M 2024-02-05T04:30:23.124 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:30:23.583 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 655 kB/s | 300 kB 00:00 2024-02-05T04:30:23.658 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 834 kB/s | 63 kB 00:00 2024-02-05T04:30:23.761 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): libcephsqlite-19.0.0-432.g3cfd0d29.el9.x 258 kB/s | 163 kB 00:00 2024-02-05T04:30:23.784 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 186 kB 00:00 2024-02-05T04:30:23.859 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 217 kB 00:00 2024-02-05T04:30:25.934 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): ceph-test-19.0.0-432.g3cfd0d29.el9.x86_6 16 MB/s | 45 MB 00:02 2024-02-05T04:30:25.937 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:30:25.938 INFO:teuthology.orchestra.run.smithi188.stdout:Total 16 MB/s | 46 MB 00:02 2024-02-05T04:30:25.938 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:30:25.958 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:30:25.958 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:30:26.312 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:30:26.312 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:30:26.862 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:30:26.936 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T04:30:26.967 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T04:30:27.018 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T04:30:27.068 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T04:30:27.090 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephsqlite-2:19.0.0-432.g3cfd0d29.el9.x86_64 5/6 2024-02-05T04:30:31.376 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephsqlite-2:19.0.0-432.g3cfd0d29.el9.x86_64 5/6 2024-02-05T04:30:31.447 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-test-2:19.0.0-432.g3cfd0d29.el9.x86_64 6/6 2024-02-05T04:30:33.677 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-test-2:19.0.0-432.g3cfd0d29.el9.x86_64 6/6 2024-02-05T04:30:33.678 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-test-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/6 2024-02-05T04:30:33.678 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephsqlite-2:19.0.0-432.g3cfd0d29.el9.x86_64 2/6 2024-02-05T04:30:33.678 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T04:30:33.678 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T04:30:33.678 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T04:30:34.092 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T04:30:34.092 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:34.092 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:30:34.092 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-test-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:34.092 INFO:teuthology.orchestra.run.smithi188.stdout: jq-1.6-16.el9.x86_64 2024-02-05T04:30:34.092 INFO:teuthology.orchestra.run.smithi188.stdout: libcephsqlite-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:34.093 INFO:teuthology.orchestra.run.smithi188.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T04:30:34.093 INFO:teuthology.orchestra.run.smithi188.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T04:30:34.093 INFO:teuthology.orchestra.run.smithi188.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T04:30:34.093 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:34.093 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:30:34.380 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph 2024-02-05T04:30:34.956 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:43 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:30:35.113 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:30:35.115 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: ceph x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 6.5 k 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 2.1 M 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 1.5 M 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 243 k 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 4.7 M 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 17 M 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T04:30:35.116 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T04:30:35.117 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T04:30:35.118 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================== 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout:Install 36 Packages 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:35.119 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 30 M 2024-02-05T04:30:35.120 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 106 M 2024-02-05T04:30:35.120 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:30:35.875 INFO:teuthology.orchestra.run.smithi188.stdout:(1/36): ceph-19.0.0-432.g3cfd0d29.el9.x86_64.rp 24 kB/s | 6.5 kB 00:00 2024-02-05T04:30:36.665 INFO:teuthology.orchestra.run.smithi188.stdout:(2/36): ceph-mgr-19.0.0-432.g3cfd0d29.el9.x86_6 1.4 MB/s | 1.5 MB 00:01 2024-02-05T04:30:37.141 INFO:teuthology.orchestra.run.smithi188.stdout:(3/36): ceph-mds-19.0.0-432.g3cfd0d29.el9.x86_6 1.4 MB/s | 2.1 MB 00:01 2024-02-05T04:30:37.274 INFO:teuthology.orchestra.run.smithi188.stdout:(4/36): ceph-mon-19.0.0-432.g3cfd0d29.el9.x86_6 3.3 MB/s | 4.7 MB 00:01 2024-02-05T04:30:37.299 INFO:teuthology.orchestra.run.smithi188.stdout:(5/36): ceph-mgr-modules-core-19.0.0-432.g3cfd0 1.5 MB/s | 243 kB 00:00 2024-02-05T04:30:37.475 INFO:teuthology.orchestra.run.smithi188.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 933 kB/s | 187 kB 00:00 2024-02-05T04:30:37.567 INFO:teuthology.orchestra.run.smithi188.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 593 kB/s | 144 kB 00:00 2024-02-05T04:30:37.592 INFO:teuthology.orchestra.run.smithi188.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 270 kB/s | 32 kB 00:00 2024-02-05T04:30:37.617 INFO:teuthology.orchestra.run.smithi188.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-02-05T04:30:37.642 INFO:teuthology.orchestra.run.smithi188.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-02-05T04:30:37.684 INFO:teuthology.orchestra.run.smithi188.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-02-05T04:30:37.776 INFO:teuthology.orchestra.run.smithi188.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 8.6 MB/s | 1.1 MB 00:00 2024-02-05T04:30:37.802 INFO:teuthology.orchestra.run.smithi188.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 883 kB/s | 103 kB 00:00 2024-02-05T04:30:37.827 INFO:teuthology.orchestra.run.smithi188.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-05T04:30:37.852 INFO:teuthology.orchestra.run.smithi188.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-05T04:30:37.878 INFO:teuthology.orchestra.run.smithi188.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-05T04:30:38.020 INFO:teuthology.orchestra.run.smithi188.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-02-05T04:30:38.586 INFO:teuthology.orchestra.run.smithi188.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 77 kB/s | 43 kB 00:00 2024-02-05T04:30:38.654 INFO:teuthology.orchestra.run.smithi188.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 195 kB/s | 151 kB 00:00 2024-02-05T04:30:38.845 INFO:teuthology.orchestra.run.smithi188.stdout:(20/36): ceph-osd-19.0.0-432.g3cfd0d29.el9.x86_ 7.7 MB/s | 17 MB 00:02 2024-02-05T04:30:38.896 INFO:teuthology.orchestra.run.smithi188.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 556 kB/s | 172 kB 00:00 2024-02-05T04:30:38.996 INFO:teuthology.orchestra.run.smithi188.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-02-05T04:30:39.013 INFO:teuthology.orchestra.run.smithi188.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-02-05T04:30:39.097 INFO:teuthology.orchestra.run.smithi188.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-02-05T04:30:39.122 INFO:teuthology.orchestra.run.smithi188.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-02-05T04:30:39.155 INFO:teuthology.orchestra.run.smithi188.stdout:(26/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 34 kB/s | 11 kB 00:00 2024-02-05T04:30:39.206 INFO:teuthology.orchestra.run.smithi188.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 180 kB/s | 20 kB 00:00 2024-02-05T04:30:39.265 INFO:teuthology.orchestra.run.smithi188.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 325 kB/s | 46 kB 00:00 2024-02-05T04:30:39.357 INFO:teuthology.orchestra.run.smithi188.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-02-05T04:30:39.381 INFO:teuthology.orchestra.run.smithi188.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-02-05T04:30:39.457 INFO:teuthology.orchestra.run.smithi188.stdout:(31/36): python3-more-itertools-8.12.0-2.el9.no 262 kB/s | 79 kB 00:00 2024-02-05T04:30:39.482 INFO:teuthology.orchestra.run.smithi188.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-02-05T04:30:39.507 INFO:teuthology.orchestra.run.smithi188.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 597 kB/s | 90 kB 00:00 2024-02-05T04:30:39.616 INFO:teuthology.orchestra.run.smithi188.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-02-05T04:30:39.675 INFO:teuthology.orchestra.run.smithi188.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2024-02-05T04:30:39.758 INFO:teuthology.orchestra.run.smithi188.stdout:(36/36): python3-webob-1.8.7-6.el9.noarch.rpm 763 kB/s | 230 kB 00:00 2024-02-05T04:30:39.764 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:30:39.765 INFO:teuthology.orchestra.run.smithi188.stdout:Total 6.4 MB/s | 30 MB 00:04 2024-02-05T04:30:39.784 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:30:39.827 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:30:39.827 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:30:40.219 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:30:40.219 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:30:41.563 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:30:41.619 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T04:30:41.651 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T04:30:41.680 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T04:30:41.740 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T04:30:41.778 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T04:30:41.812 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T04:30:41.901 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T04:30:41.972 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T04:30:42.016 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T04:30:42.084 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T04:30:42.131 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T04:30:42.170 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T04:30:42.218 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T04:30:42.273 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T04:30:42.454 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T04:30:42.507 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T04:30:42.529 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T04:30:42.564 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T04:30:42.600 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T04:30:42.836 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T04:30:42.871 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:30:42.904 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T04:30:42.944 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T04:30:42.979 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T04:30:43.014 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T04:30:43.103 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T04:30:43.137 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T04:30:43.207 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T04:30:43.345 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T04:30:43.469 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:30:43.637 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-432.g3cfd0d29.el9.n 31/36 2024-02-05T04:30:43.667 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-2:19.0.0-432.g3cfd0d29.el9.x86_64 32/36 2024-02-05T04:30:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-2:19.0.0-432.g3cfd0d29.el9.x86_64 32/36 2024-02-05T04:30:43.692 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:30:43.692 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T04:30:43.692 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:30:43.692 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T04:30:43.692 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:45.126 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-osd-2:19.0.0-432.g3cfd0d29.el9.x86_64 33/36 2024-02-05T04:30:45.152 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-osd-2:19.0.0-432.g3cfd0d29.el9.x86_64 33/36 2024-02-05T04:30:45.153 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:30:45.153 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T04:30:45.153 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:30:45.153 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T04:30:45.153 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:45.611 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mon-2:19.0.0-432.g3cfd0d29.el9.x86_64 34/36 2024-02-05T04:30:45.636 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mon-2:19.0.0-432.g3cfd0d29.el9.x86_64 34/36 2024-02-05T04:30:45.636 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:30:45.636 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T04:30:45.636 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:30:45.636 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T04:30:45.636 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:45.853 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mds-2:19.0.0-432.g3cfd0d29.el9.x86_64 35/36 2024-02-05T04:30:45.877 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mds-2:19.0.0-432.g3cfd0d29.el9.x86_64 35/36 2024-02-05T04:30:45.877 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:30:45.877 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T04:30:45.877 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:30:45.877 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T04:30:45.877 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:45.952 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-2:19.0.0-432.g3cfd0d29.el9.x86_64 36/36 2024-02-05T04:30:47.651 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-2:19.0.0-432.g3cfd0d29.el9.x86_64 36/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mds-2:19.0.0-432.g3cfd0d29.el9.x86_64 2/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-2:19.0.0-432.g3cfd0d29.el9.x86_64 3/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mon-2:19.0.0-432.g3cfd0d29.el9.x86_64 4/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-osd-2:19.0.0-432.g3cfd0d29.el9.x86_64 5/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-432.g3cfd0d29.el9.n 6/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T04:30:47.652 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T04:30:47.653 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T04:30:47.655 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T04:30:48.772 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T04:30:48.772 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mds-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-modules-core-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mon-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-osd-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T04:30:48.773 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T04:30:48.774 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:48.775 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:30:49.144 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-base 2024-02-05T04:30:49.711 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:58 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:30:49.814 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-base-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:30:49.866 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:30:49.867 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:30:49.867 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:30:49.929 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install cephadm 2024-02-05T04:30:50.495 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:00:59 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:30:50.649 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:30:50.650 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:50.650 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:30:50.650 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:50.650 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:30:50.650 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 806 k 2024-02-05T04:30:50.650 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:50.650 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:30:50.651 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:50.651 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:30:50.651 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:50.651 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 806 k 2024-02-05T04:30:50.651 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 823 k 2024-02-05T04:30:50.651 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:30:51.844 INFO:teuthology.orchestra.run.smithi188.stdout:cephadm-19.0.0-432.g3cfd0d29.el9.noarch.rpm 677 kB/s | 806 kB 00:01 2024-02-05T04:30:51.844 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:30:51.844 INFO:teuthology.orchestra.run.smithi188.stdout:Total 675 kB/s | 806 kB 00:01 2024-02-05T04:30:51.845 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:30:51.851 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:30:51.851 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:30:51.856 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:30:51.857 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:30:51.931 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:30:52.349 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 1/1 2024-02-05T04:30:52.441 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 1/1 2024-02-05T04:30:52.827 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 1/1 2024-02-05T04:30:53.180 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 1/1 2024-02-05T04:30:53.180 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:53.181 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:30:53.181 INFO:teuthology.orchestra.run.smithi188.stdout: cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:30:53.181 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:53.181 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:30:53.381 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T04:30:53.961 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:02 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:30:54.117 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:30:54.118 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:54.118 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-02-05T04:30:54.118 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 145 k 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 145 k 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 436 k 2024-02-05T04:30:54.119 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:30:54.799 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-immutable-object-cache-19.0.0-432.g3cfd0d2 214 kB/s | 145 kB 00:00 2024-02-05T04:30:54.800 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:30:54.800 INFO:teuthology.orchestra.run.smithi188.stdout:Total 213 kB/s | 145 kB 00:00 2024-02-05T04:30:54.800 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:30:54.808 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:30:54.808 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:30:54.862 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:30:54.862 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:30:55.025 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:30:55.092 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-432.g3cfd0d29.e 1/1 2024-02-05T04:30:55.116 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-432.g3cfd0d29.e 1/1 2024-02-05T04:30:55.116 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:30:55.116 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T04:30:55.117 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:56.122 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-432.g3cfd0d29.e 1/1 2024-02-05T04:30:56.122 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:56.122 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:30:56.122 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-immutable-object-cache-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:30:56.122 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:56.122 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:30:56.363 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr 2024-02-05T04:30:56.948 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:05 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:30:57.050 INFO:teuthology.orchestra.run.smithi188.stdout:Package ceph-mgr-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:30:57.100 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:30:57.101 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:30:57.101 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:30:57.166 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T04:30:57.730 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:06 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:30:57.885 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:30:57.886 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:57.886 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:30:57.886 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:57.886 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:30:57.886 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 1.8 M 2024-02-05T04:30:57.886 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:30:57.886 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 28 k 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 15 k 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout:Install 5 Packages 2024-02-05T04:30:57.887 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:30:57.888 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 2.0 M 2024-02-05T04:30:57.888 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 8.6 M 2024-02-05T04:30:57.888 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:30:58.329 INFO:teuthology.orchestra.run.smithi188.stdout:(1/5): ceph-grafana-dashboards-19.0.0-432.g3cfd 147 kB/s | 28 kB 00:00 2024-02-05T04:30:58.529 INFO:teuthology.orchestra.run.smithi188.stdout:(2/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 154 kB/s | 31 kB 00:00 2024-02-05T04:30:58.554 INFO:teuthology.orchestra.run.smithi188.stdout:(3/5): ceph-prometheus-alerts-19.0.0-432.g3cfd0 36 kB/s | 15 kB 00:00 2024-02-05T04:30:58.696 INFO:teuthology.orchestra.run.smithi188.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.1 MB/s | 188 kB 00:00 2024-02-05T04:30:58.896 INFO:teuthology.orchestra.run.smithi188.stdout:(5/5): ceph-mgr-dashboard-19.0.0-432.g3cfd0d29. 2.3 MB/s | 1.8 MB 00:00 2024-02-05T04:30:58.898 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:30:58.899 INFO:teuthology.orchestra.run.smithi188.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-02-05T04:30:58.899 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:30:58.906 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:30:58.906 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:30:58.975 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:30:58.975 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:30:59.218 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:30:59.302 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-02-05T04:30:59.334 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-02-05T04:30:59.368 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-432.g3cfd0d29.el9.no 3/5 2024-02-05T04:30:59.607 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-432.g3cfd0d29.el9.n 4/5 2024-02-05T04:30:59.636 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-432.g3cfd0d29.el9.noarch 5/5 2024-02-05T04:31:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-432.g3cfd0d29.el9.noarch 5/5 2024-02-05T04:31:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-432.g3cfd0d29.el9.n 1/5 2024-02-05T04:31:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-432.g3cfd0d29.el9.noarch 2/5 2024-02-05T04:31:00.215 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-432.g3cfd0d29.el9.no 3/5 2024-02-05T04:31:00.216 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-02-05T04:31:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-02-05T04:31:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:00.505 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-grafana-dashboards-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:31:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-dashboard-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:31:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-prometheus-alerts-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:31:00.505 INFO:teuthology.orchestra.run.smithi188.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T04:31:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T04:31:00.506 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:00.506 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:00.772 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T04:31:01.334 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:10 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:01.492 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:01.493 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================== 2024-02-05T04:31:01.493 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:31:01.493 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================== 2024-02-05T04:31:01.493 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:01.493 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 7.4 M 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:01.494 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:01.495 INFO:teuthology.orchestra.run.smithi188.stdout:=========================================================================================== 2024-02-05T04:31:01.495 INFO:teuthology.orchestra.run.smithi188.stdout:Install 12 Packages 2024-02-05T04:31:01.495 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:01.495 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 37 M 2024-02-05T04:31:01.495 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 209 M 2024-02-05T04:31:01.495 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:01.781 INFO:teuthology.orchestra.run.smithi188.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 129 kB/s | 32 kB 00:00 2024-02-05T04:31:01.848 INFO:teuthology.orchestra.run.smithi188.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 256 kB/s | 17 kB 00:00 2024-02-05T04:31:01.948 INFO:teuthology.orchestra.run.smithi188.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 342 kB/s | 34 kB 00:00 2024-02-05T04:31:02.132 INFO:teuthology.orchestra.run.smithi188.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-02-05T04:31:02.599 INFO:teuthology.orchestra.run.smithi188.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.3 MB/s | 4.7 MB 00:00 2024-02-05T04:31:02.666 INFO:teuthology.orchestra.run.smithi188.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.0 MB/s | 4.8 MB 00:00 2024-02-05T04:31:02.691 INFO:teuthology.orchestra.run.smithi188.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-02-05T04:31:03.083 INFO:teuthology.orchestra.run.smithi188.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 802 kB 00:00 2024-02-05T04:31:03.483 INFO:teuthology.orchestra.run.smithi188.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-02-05T04:31:03.508 INFO:teuthology.orchestra.run.smithi188.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 458 kB/s | 194 kB 00:00 2024-02-05T04:31:04.225 INFO:teuthology.orchestra.run.smithi188.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 331 kB/s | 245 kB 00:00 2024-02-05T04:31:05.842 INFO:teuthology.orchestra.run.smithi188.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-4 1.7 MB/s | 7.4 MB 00:04 2024-02-05T04:31:05.846 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:05.846 INFO:teuthology.orchestra.run.smithi188.stdout:Total 8.5 MB/s | 37 MB 00:04 2024-02-05T04:31:05.860 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:05.887 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:05.887 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:06.263 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:06.264 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:06.955 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:07.027 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T04:31:07.062 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T04:31:07.160 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:31:07.208 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T04:31:07.385 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:31:07.415 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:31:07.496 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T04:31:08.153 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T04:31:08.253 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T04:31:09.485 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T04:31:10.511 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T04:31:10.540 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-432.g3cfd0d 12/12 2024-02-05T04:31:13.110 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-432.g3cfd0d 12/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-432.g3cfd0d 1/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T04:31:13.111 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-diskprediction-local-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T04:31:13.563 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T04:31:13.564 INFO:teuthology.orchestra.run.smithi188.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T04:31:13.567 INFO:teuthology.orchestra.run.smithi188.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T04:31:13.567 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T04:31:13.567 INFO:teuthology.orchestra.run.smithi188.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T04:31:13.567 INFO:teuthology.orchestra.run.smithi188.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T04:31:13.567 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:13.567 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:13.946 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-rook 2024-02-05T04:31:14.540 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:23 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:14.709 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 50 k 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T04:31:14.710 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout:Install 13 Packages 2024-02-05T04:31:14.711 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:14.712 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 2.0 M 2024-02-05T04:31:14.712 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 26 M 2024-02-05T04:31:14.712 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:15.215 INFO:teuthology.orchestra.run.smithi188.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2024-02-05T04:31:15.240 INFO:teuthology.orchestra.run.smithi188.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 97 kB/s | 18 kB 00:00 2024-02-05T04:31:15.265 INFO:teuthology.orchestra.run.smithi188.stdout:(3/13): ceph-mgr-rook-19.0.0-432.g3cfd0d29.el9. 237 kB/s | 50 kB 00:00 2024-02-05T04:31:15.366 INFO:teuthology.orchestra.run.smithi188.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-02-05T04:31:15.408 INFO:teuthology.orchestra.run.smithi188.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 790 kB/s | 132 kB 00:00 2024-02-05T04:31:15.433 INFO:teuthology.orchestra.run.smithi188.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 211 kB 00:00 2024-02-05T04:31:15.467 INFO:teuthology.orchestra.run.smithi188.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 426 kB/s | 43 kB 00:00 2024-02-05T04:31:15.617 INFO:teuthology.orchestra.run.smithi188.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 100 kB/s | 15 kB 00:00 2024-02-05T04:31:15.651 INFO:teuthology.orchestra.run.smithi188.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-02-05T04:31:15.709 INFO:teuthology.orchestra.run.smithi188.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 838 kB/s | 203 kB 00:00 2024-02-05T04:31:15.818 INFO:teuthology.orchestra.run.smithi188.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 5.1 MB/s | 1.0 MB 00:00 2024-02-05T04:31:15.844 INFO:teuthology.orchestra.run.smithi188.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 307 kB/s | 59 kB 00:00 2024-02-05T04:31:15.868 INFO:teuthology.orchestra.run.smithi188.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 565 kB/s | 90 kB 00:00 2024-02-05T04:31:15.873 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:15.873 INFO:teuthology.orchestra.run.smithi188.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-02-05T04:31:15.873 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:15.890 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:15.890 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:15.989 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:15.989 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:16.507 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:16.673 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T04:31:16.727 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T04:31:16.783 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T04:31:16.821 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T04:31:16.862 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T04:31:16.955 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T04:31:17.138 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T04:31:17.213 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T04:31:17.926 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T04:31:17.985 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T04:31:18.019 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T04:31:18.066 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T04:31:18.089 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-rook-2:19.0.0-432.g3cfd0d29.el9.noarch 13/13 2024-02-05T04:31:18.760 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-432.g3cfd0d29.el9.noarch 13/13 2024-02-05T04:31:18.760 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-rook-2:19.0.0-432.g3cfd0d29.el9.noarch 1/13 2024-02-05T04:31:18.760 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T04:31:18.761 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T04:31:19.132 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T04:31:19.132 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:19.132 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-rook-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:19.133 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:19.399 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T04:31:19.971 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:28 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:20.128 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:20.128 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-02-05T04:31:20.128 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:31:20.128 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 133 k 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:20.129 INFO:teuthology.orchestra.run.smithi188.stdout:======================================================================================= 2024-02-05T04:31:20.130 INFO:teuthology.orchestra.run.smithi188.stdout:Install 6 Packages 2024-02-05T04:31:20.130 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:20.130 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 6.8 M 2024-02-05T04:31:20.130 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 33 M 2024-02-05T04:31:20.130 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:20.646 INFO:teuthology.orchestra.run.smithi188.stdout:(1/6): ceph-mgr-cephadm-19.0.0-432.g3cfd0d29.el 521 kB/s | 133 kB 00:00 2024-02-05T04:31:20.779 INFO:teuthology.orchestra.run.smithi188.stdout:(2/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 589 kB/s | 228 kB 00:00 2024-02-05T04:31:21.138 INFO:teuthology.orchestra.run.smithi188.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 7.7 MB/s | 5.8 MB 00:00 2024-02-05T04:31:21.405 INFO:teuthology.orchestra.run.smithi188.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 722 kB/s | 548 kB 00:00 2024-02-05T04:31:21.430 INFO:teuthology.orchestra.run.smithi188.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 89 kB/s | 58 kB 00:00 2024-02-05T04:31:21.455 INFO:teuthology.orchestra.run.smithi188.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2024-02-05T04:31:21.458 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:21.459 INFO:teuthology.orchestra.run.smithi188.stdout:Total 5.1 MB/s | 6.8 MB 00:01 2024-02-05T04:31:21.459 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:21.467 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:21.467 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:21.557 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:21.558 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:21.855 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:22.009 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T04:31:22.065 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T04:31:22.583 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T04:31:22.659 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T04:31:22.719 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T04:31:22.744 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 6/6 2024-02-05T04:31:23.554 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 6/6 2024-02-05T04:31:23.555 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 1/6 2024-02-05T04:31:23.555 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T04:31:23.555 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T04:31:23.555 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T04:31:23.555 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T04:31:23.867 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T04:31:23.867 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:23.867 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:23.867 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-mgr-cephadm-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:31:23.867 INFO:teuthology.orchestra.run.smithi188.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T04:31:23.867 INFO:teuthology.orchestra.run.smithi188.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T04:31:23.868 INFO:teuthology.orchestra.run.smithi188.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T04:31:23.868 INFO:teuthology.orchestra.run.smithi188.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T04:31:23.868 INFO:teuthology.orchestra.run.smithi188.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T04:31:23.868 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:23.868 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:24.134 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-fuse 2024-02-05T04:31:24.699 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:33 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:24.858 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:24.859 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:24.859 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:31:24.859 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:24.859 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:24.859 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 886 k 2024-02-05T04:31:24.859 INFO:teuthology.orchestra.run.smithi188.stdout:Installing dependencies: 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:Install 2 Packages 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 964 k 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 2.8 M 2024-02-05T04:31:24.860 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:25.231 INFO:teuthology.orchestra.run.smithi188.stdout:(1/2): ceph-fuse-19.0.0-432.g3cfd0d29.el9.x86_6 2.3 MB/s | 886 kB 00:00 2024-02-05T04:31:25.990 INFO:teuthology.orchestra.run.smithi188.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 70 kB/s | 79 kB 00:01 2024-02-05T04:31:25.991 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:25.992 INFO:teuthology.orchestra.run.smithi188.stdout:Total 851 kB/s | 964 kB 00:01 2024-02-05T04:31:25.992 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:26.013 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:26.013 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:26.088 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:26.089 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:26.278 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:26.432 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T04:31:26.498 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 2/2 2024-02-05T04:31:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 2/2 2024-02-05T04:31:27.196 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/2 2024-02-05T04:31:27.514 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T04:31:27.514 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:27.514 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:27.514 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T04:31:27.514 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:27.514 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:27.783 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install ceph-volume 2024-02-05T04:31:28.360 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:37 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:28.516 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:28.517 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:28.517 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repository Size 2024-02-05T04:31:28.517 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:28.517 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:28.517 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume noarch 2:19.0.0-432.g3cfd0d29.el9 ceph-noarch 262 k 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 262 k 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 1.2 M 2024-02-05T04:31:28.518 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:29.403 INFO:teuthology.orchestra.run.smithi188.stdout:ceph-volume-19.0.0-432.g3cfd0d29.el9.noarch.rpm 296 kB/s | 262 kB 00:00 2024-02-05T04:31:29.404 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:29.404 INFO:teuthology.orchestra.run.smithi188.stdout:Total 296 kB/s | 262 kB 00:00 2024-02-05T04:31:29.404 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:29.412 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:29.412 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:29.452 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:29.452 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:29.669 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:29.747 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : ceph-volume-2:19.0.0-432.g3cfd0d29.el9.noarch 1/1 2024-02-05T04:31:29.762 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: ceph-volume-2:19.0.0-432.g3cfd0d29.el9.noarch 1/1 2024-02-05T04:31:29.762 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:31:29.762 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T04:31:29.762 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:30.843 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : ceph-volume-2:19.0.0-432.g3cfd0d29.el9.noarch 1/1 2024-02-05T04:31:30.843 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:30.843 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:30.843 INFO:teuthology.orchestra.run.smithi188.stdout: ceph-volume-2:19.0.0-432.g3cfd0d29.el9.noarch 2024-02-05T04:31:30.843 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:30.843 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:31.152 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados-devel 2024-02-05T04:31:31.737 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:31.895 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 127 k 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:31.896 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:31:31.897 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:31.897 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 127 k 2024-02-05T04:31:31.897 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 456 k 2024-02-05T04:31:31.897 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:32.642 INFO:teuthology.orchestra.run.smithi188.stdout:librados-devel-19.0.0-432.g3cfd0d29.el9.x86_64. 170 kB/s | 127 kB 00:00 2024-02-05T04:31:32.642 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:32.642 INFO:teuthology.orchestra.run.smithi188.stdout:Total 170 kB/s | 127 kB 00:00 2024-02-05T04:31:32.643 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:32.648 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:32.649 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:32.707 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:32.708 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:32.857 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:32.956 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : librados-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:33.396 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: librados-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:33.692 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : librados-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:33.692 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:33.692 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:33.692 INFO:teuthology.orchestra.run.smithi188.stdout: librados-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:31:33.692 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:33.692 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:33.936 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs2 2024-02-05T04:31:34.515 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:34.624 INFO:teuthology.orchestra.run.smithi188.stdout:Package libcephfs2-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:31:34.678 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:34.679 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:31:34.679 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:34.741 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install libcephfs-devel 2024-02-05T04:31:35.311 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:35.466 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 31 k 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:35.467 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 31 k 2024-02-05T04:31:35.468 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 140 k 2024-02-05T04:31:35.468 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:35.753 INFO:teuthology.orchestra.run.smithi188.stdout:libcephfs-devel-19.0.0-432.g3cfd0d29.el9.x86_64 109 kB/s | 31 kB 00:00 2024-02-05T04:31:35.754 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:35.754 INFO:teuthology.orchestra.run.smithi188.stdout:Total 109 kB/s | 31 kB 00:00 2024-02-05T04:31:35.754 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:35.757 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:35.758 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:35.786 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:35.787 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:35.882 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:35.994 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : libcephfs-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:36.394 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:36.664 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : libcephfs-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:36.664 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:36.665 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:36.665 INFO:teuthology.orchestra.run.smithi188.stdout: libcephfs-devel-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:31:36.665 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:36.665 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:36.853 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librados2 2024-02-05T04:31:37.420 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:46 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:37.527 INFO:teuthology.orchestra.run.smithi188.stdout:Package librados2-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:31:37.578 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:37.579 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:31:37.579 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:37.653 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install librbd1 2024-02-05T04:31:38.226 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:47 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:38.330 INFO:teuthology.orchestra.run.smithi188.stdout:Package librbd1-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:31:38.382 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:38.383 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:31:38.383 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:38.449 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rados 2024-02-05T04:31:39.019 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:48 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:39.123 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rados-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:31:39.176 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:39.177 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:31:39.177 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:39.241 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rgw 2024-02-05T04:31:39.814 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:48 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:39.919 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rgw-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:31:39.972 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:39.972 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:31:39.972 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:40.036 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-cephfs 2024-02-05T04:31:40.607 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:49 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:40.711 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-cephfs-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:31:40.763 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:40.764 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:31:40.764 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:40.834 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install python3-rbd 2024-02-05T04:31:41.403 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:50 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:41.508 INFO:teuthology.orchestra.run.smithi188.stdout:Package python3-rbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 is already installed. 2024-02-05T04:31:41.562 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:41.563 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do. 2024-02-05T04:31:41.563 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:41.628 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-fuse 2024-02-05T04:31:42.199 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:51 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:42.357 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 85 k 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:31:42.358 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:42.359 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 85 k 2024-02-05T04:31:42.359 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 231 k 2024-02-05T04:31:42.359 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:42.772 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-fuse-19.0.0-432.g3cfd0d29.el9.x86_64.rpm 206 kB/s | 85 kB 00:00 2024-02-05T04:31:42.772 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:42.773 INFO:teuthology.orchestra.run.smithi188.stdout:Total 206 kB/s | 85 kB 00:00 2024-02-05T04:31:42.773 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:42.780 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:42.780 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:42.834 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:42.834 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:42.973 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:43.072 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:43.477 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:43.792 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:43.792 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:43.792 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:43.792 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-fuse-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:31:43.792 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:43.792 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:44.008 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-mirror 2024-02-05T04:31:44.603 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:53 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:44.772 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:44.773 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:44.773 INFO:teuthology.orchestra.run.smithi188.stdout: Package Arch Version Repo Size 2024-02-05T04:31:44.773 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:44.773 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 3.1 M 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 3.1 M 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 13 M 2024-02-05T04:31:44.774 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:45.186 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-mirror-19.0.0-432.g3cfd0d29.el9.x86_64.rpm 7.5 MB/s | 3.1 MB 00:00 2024-02-05T04:31:45.187 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:45.187 INFO:teuthology.orchestra.run.smithi188.stdout:Total 7.5 MB/s | 3.1 MB 00:00 2024-02-05T04:31:45.188 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:45.197 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:45.197 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:45.272 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:45.272 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:45.704 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:45.762 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-mirror-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-mirror-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:45.788 INFO:teuthology.orchestra.run.smithi188.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T04:31:45.788 INFO:teuthology.orchestra.run.smithi188.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T04:31:45.788 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:31:45.788 INFO:teuthology.orchestra.run.smithi188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T04:31:45.788 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:47.063 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-mirror-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:47.147 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:47.148 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:47.148 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-mirror-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:31:47.148 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:47.148 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:47.342 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum -y install rbd-nbd 2024-02-05T04:31:47.927 INFO:teuthology.orchestra.run.smithi188.stdout:Last metadata expiration check: 0:01:56 ago on Mon 05 Feb 2024 04:29:51 AM UTC. 2024-02-05T04:31:48.082 INFO:teuthology.orchestra.run.smithi188.stdout:Dependencies resolved. 2024-02-05T04:31:48.083 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: Package Architecture Version Repository Size 2024-02-05T04:31:48.083 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:48.083 INFO:teuthology.orchestra.run.smithi188.stdout:Installing: 2024-02-05T04:31:48.083 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd x86_64 2:19.0.0-432.g3cfd0d29.el9 ceph 170 k 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction Summary 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout:================================================================================ 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout:Install 1 Package 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout:Total download size: 170 k 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout:Installed size: 490 k 2024-02-05T04:31:48.084 INFO:teuthology.orchestra.run.smithi188.stdout:Downloading Packages: 2024-02-05T04:31:48.648 INFO:teuthology.orchestra.run.smithi188.stdout:rbd-nbd-19.0.0-432.g3cfd0d29.el9.x86_64.rpm 302 kB/s | 170 kB 00:00 2024-02-05T04:31:48.649 INFO:teuthology.orchestra.run.smithi188.stdout:-------------------------------------------------------------------------------- 2024-02-05T04:31:48.649 INFO:teuthology.orchestra.run.smithi188.stdout:Total 301 kB/s | 170 kB 00:00 2024-02-05T04:31:48.649 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction check 2024-02-05T04:31:48.657 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction check succeeded. 2024-02-05T04:31:48.657 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction test 2024-02-05T04:31:48.713 INFO:teuthology.orchestra.run.smithi188.stdout:Transaction test succeeded. 2024-02-05T04:31:48.713 INFO:teuthology.orchestra.run.smithi188.stdout:Running transaction 2024-02-05T04:31:48.863 INFO:teuthology.orchestra.run.smithi188.stdout: Preparing : 1/1 2024-02-05T04:31:48.954 INFO:teuthology.orchestra.run.smithi188.stdout: Installing : rbd-nbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:49.404 INFO:teuthology.orchestra.run.smithi188.stdout: Running scriptlet: rbd-nbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:49.724 INFO:teuthology.orchestra.run.smithi188.stdout: Verifying : rbd-nbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 1/1 2024-02-05T04:31:49.724 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:49.725 INFO:teuthology.orchestra.run.smithi188.stdout:Installed: 2024-02-05T04:31:49.725 INFO:teuthology.orchestra.run.smithi188.stdout: rbd-nbd-2:19.0.0-432.g3cfd0d29.el9.x86_64 2024-02-05T04:31:49.725 INFO:teuthology.orchestra.run.smithi188.stdout: 2024-02-05T04:31:49.725 INFO:teuthology.orchestra.run.smithi188.stdout:Complete! 2024-02-05T04:31:49.969 DEBUG:teuthology.parallel:result is None 2024-02-05T04:31:49.969 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-05T04:31:50.389 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T04:31:50.412 INFO:teuthology.orchestra.run.smithi188.stdout:19.0.0-432.g3cfd0d29.el9 2024-02-05T04:31:50.413 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-432.g3cfd0d29.el9 2024-02-05T04:31:50.413 INFO:teuthology.task.install:The correct ceph version 19.0.0-432.g3cfd0d29 is installed. 2024-02-05T04:31:50.415 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-05T04:31:50.415 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-02-05T04:31:50.416 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T04:31:50.482 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-05T04:31:50.482 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-02-05T04:31:50.482 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T04:31:50.549 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T04:31:50.616 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-05T04:31:50.617 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-02-05T04:31:50.617 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T04:31:50.683 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T04:31:50.750 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-05T04:31:50.750 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-02-05T04:31:50.750 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T04:31:50.820 DEBUG:teuthology.orchestra.run.smithi188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T04:31:50.887 INFO:teuthology.run_tasks:Running task workunit... 2024-02-05T04:31:50.902 INFO:tasks.workunit:Pulling workunits from ref 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-05T04:31:50.902 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-05T04:31:50.902 INFO:tasks.workunit:timeout=3h 2024-02-05T04:31:50.902 INFO:tasks.workunit:cleanup=True 2024-02-05T04:31:50.902 DEBUG:teuthology.orchestra.run.smithi188:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-05T04:31:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T04:31:50.943 INFO:teuthology.orchestra.run.smithi188.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-02-05T04:31:50.943 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-02-05T04:31:50.998 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-02-05T04:31:50.998 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-02-05T04:31:51.053 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 2024-02-05T04:31:51.108 INFO:tasks.workunit.client.0.smithi188.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-05T04:33:05.719 INFO:tasks.workunit.client.0.smithi188.stderr:Note: switching to '3cfd0d29cc3905ea860fc5482a5a58407e15a0d7'. 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr:state without impacting any branches by switching back to a branch. 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: git switch -c 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr:Or undo this operation with: 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: git switch - 2024-02-05T04:33:05.720 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-02-05T04:33:05.721 INFO:tasks.workunit.client.0.smithi188.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-05T04:33:05.721 INFO:tasks.workunit.client.0.smithi188.stderr: 2024-02-05T04:33:05.721 INFO:tasks.workunit.client.0.smithi188.stderr:HEAD is now at 3cfd0d29cc3 os/bluestore: use blob_offset in compressed read results 2024-02-05T04:33:05.723 DEBUG:teuthology.orchestra.run.smithi188:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-05T04:33:05.779 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2024-02-05T04:33:05.779 DEBUG:teuthology.orchestra.run.smithi188:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-05T04:33:05.835 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-02-05T04:33:05.835 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-02-05T04:33:05.836 DEBUG:teuthology.orchestra.run.smithi188:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-02-05T04:33:05.906 INFO:tasks.workunit.client.0.smithi188.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T04:33:05.911 INFO:tasks.workunit.client.0.smithi188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T04:33:05.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:33:05.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:33:05.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T04:33:05.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-02-05T04:33:05.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-02-05T04:33:05.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none ' 2024-02-05T04:33:05.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-02-05T04:33:05.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T04:33:05.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-02-05T04:33:05.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:33:05.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:33:05.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:33:05.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:33:05.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:33:05.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:33:05.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:33:05.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:33:05.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:33:05.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:33:05.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:33:05.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:33:05.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:33:05.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:33:05.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:33:05.924 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:33:05.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:33:05.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:33:05.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-05T04:33:05.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:33:05.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:33:05.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:33:05.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27892 2024-02-05T04:33:05.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:33:05.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:33:05.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-02-05T04:33:05.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:33:05.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:33:05.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:33:05.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27892 2024-02-05T04:33:05.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:33:05.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-02-05T04:33:05.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-02-05T04:33:05.935 INFO:tasks.workunit.client.0.smithi188.stdout:Dir: td/osd-mapper 2024-02-05T04:33:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-02-05T04:33:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-02-05T04:33:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-02-05T04:33:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-02-05T04:33:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-02-05T04:33:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-02-05T04:33:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-02-05T04:33:05.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-02-05T04:33:05.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-02-05T04:33:05.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-02-05T04:33:05.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-02-05T04:33:07.498 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 616cffa0-c8ee-445b-9bff-8792133f02f8 2024-02-05T04:33:07.766 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:33:07.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:07.836+0000 7fb00e6104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:07.838 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:07.838+0000 7fb00e6104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:07.839 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:07.839+0000 7fb00e6104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:07.839 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:07.839+0000 7fb00e6104c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-05T04:33:07.839 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:07.839+0000 7fb00e6104c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-05T04:33:07.840 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:07.839+0000 7fb00e6104c0 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-02-05T04:33:07.919 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:07.918+0000 7fb00e6104c0 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-02-05T04:33:13.732 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:33:14.067 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:33:14.094 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:14.094+0000 7f75c23154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:14.096 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:14.095+0000 7f75c23154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:14.097 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:14.096+0000 7f75c23154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:14.331 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:33:14.616 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:14.616+0000 7f75c23154c0 -1 Falling back to public interface 2024-02-05T04:33:15.601 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:33:16.876 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:33:16.998 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:16.998+0000 7f75c23154c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:33:18.174 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:33:19.452 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:33:20.731 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:33:22.006 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:33:23.284 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:33:24.562 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:33:25.941 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:33:27.217 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:33:28.717 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:33:30.556 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:33:32.174 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:33:33.751 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:33:35.310 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:33:37.109 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:33:39.259 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:33:41.020 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:33:41.293 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1777826620,v1:127.0.0.1:6803/1777826620] [v2:127.0.0.1:6804/1777826620,v1:127.0.0.1:6805/1777826620] exists,up 616cffa0-c8ee-445b-9bff-8792133f02f8 2024-02-05T04:33:41.296 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 1634d1c8-6675-4025-9447-c3a86149ff8a 2024-02-05T04:33:41.689 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:33:41.730 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:41.729+0000 7fd8cfc0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:41.776 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:41.775+0000 7fd8cfc0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:41.776 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:41.776+0000 7fd8cfc0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:41.777 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:41.776+0000 7fd8cfc0d4c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-05T04:33:41.777 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:41.776+0000 7fd8cfc0d4c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-05T04:33:41.777 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:41.776+0000 7fd8cfc0d4c0 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-02-05T04:33:41.832 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:41.832+0000 7fd8cfc0d4c0 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-02-05T04:33:47.716 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:33:48.056 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:33:48.083 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:48.083+0000 7f95d7d254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:48.085 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:48.085+0000 7f95d7d254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:48.086 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:48.086+0000 7f95d7d254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:33:48.325 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:33:48.614 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:48.614+0000 7f95d7d254c0 -1 Falling back to public interface 2024-02-05T04:33:49.625 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:33:50.901 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:33:50.951 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:33:50.950+0000 7f95d7d254c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:33:52.173 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:33:53.451 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:33:54.728 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:33:56.055 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:33:57.329 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:33:58.603 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:33:59.924 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:34:01.233 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:34:02.526 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:34:04.457 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:34:05.885 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:34:07.285 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:34:08.846 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:34:10.312 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:34:11.895 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:34:13.479 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:34:13.770 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2474276275,v1:127.0.0.1:6811/2474276275] [v2:127.0.0.1:6812/2474276275,v1:127.0.0.1:6813/2474276275] exists,up 1634d1c8-6675-4025-9447-c3a86149ff8a 2024-02-05T04:34:13.774 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 9e818ca3-45b3-472f-9078-832eacbe84d1 2024-02-05T04:34:14.171 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:34:14.212 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:14.212+0000 7f79d3a114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:34:14.223 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:14.222+0000 7f79d3a114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:34:14.224 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:14.223+0000 7f79d3a114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:34:14.224 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:14.224+0000 7f79d3a114c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-05T04:34:14.224 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:14.224+0000 7f79d3a114c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-05T04:34:14.224 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:14.224+0000 7f79d3a114c0 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-02-05T04:34:14.272 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:14.273+0000 7f79d3a114c0 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-02-05T04:34:20.147 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:34:20.489 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:34:20.516 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:20.515+0000 7fcd63f8b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:34:20.518 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:20.518+0000 7fcd63f8b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:34:20.519 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:20.519+0000 7fcd63f8b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:34:20.755 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:34:21.039 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:21.039+0000 7fcd63f8b4c0 -1 Falling back to public interface 2024-02-05T04:34:22.019 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:34:23.292 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:34:23.371 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:34:23.370+0000 7fcd63f8b4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:34:24.579 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:34:25.855 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:34:27.158 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:34:28.432 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:34:29.706 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:34:31.031 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:34:32.351 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:34:33.628 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:34:35.091 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:34:36.445 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:34:37.719 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:34:39.074 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:34:40.693 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:34:42.363 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:34:43.741 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:34:45.075 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:34:46.645 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T04:34:46.920 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1784918958,v1:127.0.0.1:6819/1784918958] [v2:127.0.0.1:6820/1784918958,v1:127.0.0.1:6821/1784918958] exists,up 9e818ca3-45b3-472f-9078-832eacbe84d1 2024-02-05T04:34:47.297 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T04:34:49.073 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836499 2024-02-05T04:34:51.891 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672972 2024-02-05T04:34:52.157 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509444 2024-02-05T04:34:54.611 INFO:tasks.workunit.client.0.smithi188.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-02-05T04:34:54.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-02-05T04:34:54.703 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T04:34:54.703 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:34:54.704 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:54.718 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T04:34:54.718 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:34:54.718 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:54.731 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2: { 2024-02-05T04:34:54.731 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:34:54.731 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:54.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-02-05T04:34:54.851 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T04:34:54.851 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:34:54.851 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:54.866 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T04:34:54.866 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:34:54.866 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:54.880 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2: { 2024-02-05T04:34:54.880 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-02-05T04:34:54.880 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:54.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-02-05T04:34:54.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-02-05T04:34:54.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-02-05T04:34:54.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-02-05T04:34:54.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-02-05T04:34:54.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-05T04:34:54.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-05T04:34:55.279 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap01 2024-02-05T04:34:55.630 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap02 2024-02-05T04:34:55.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-02-05T04:34:55.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-05T04:34:55.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-05T04:34:55.992 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap13 2024-02-05T04:34:55.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-02-05T04:34:55.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-02-05T04:34:55.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-02-05T04:34:56.403 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap24 2024-02-05T04:34:56.637 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap25 2024-02-05T04:34:56.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 20953 2024-02-05T04:34:56.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-02-05T04:34:56.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-05T04:34:56.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-02-05T04:34:57.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-02-05T04:34:57.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-05T04:34:57.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-02-05T04:34:57.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-02-05T04:34:57.482 INFO:tasks.workunit.client.0.smithi188.stdout:pgid is 1.3 (primary: osd.1) 2024-02-05T04:34:57.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-02-05T04:34:57.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-02-05T04:34:57.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:294: set_query_debug: local pgid=1.3 2024-02-05T04:34:57.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: ceph pg dump pgs_brief 2024-02-05T04:34:57.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-02-05T04:34:57.731 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs_brief 2024-02-05T04:34:57.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: local prim_osd=1 2024-02-05T04:34:57.742 INFO:tasks.workunit.client.0.smithi188.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-02-05T04:34:57.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-02-05T04:34:57.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: get_asok_path osd.1 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27892/ceph-osd.1.asok 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: CEPH_ARGS= 2024-02-05T04:34:57.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.27892/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-02-05T04:34:57.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:34:57.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout:{"success":true}{ 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "objxxx", 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 2, 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [ 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 1, 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap01" 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 2, 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap02" 2024-02-05T04:34:57.861 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 6, 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "overlaps": [] 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 3, 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [ 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 3, 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap13" 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 5, 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: "overlaps": [] 2024-02-05T04:34:57.862 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 5, 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [ 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 4, 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap24" 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 5, 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap25" 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 6, 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: "overlaps": [] 2024-02-05T04:34:57.863 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stdout: "id": "head", 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [], 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 6 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:57.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-05T04:34:57.953 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:34:57.954 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T04:34:57.954 INFO:tasks.workunit.client.0.smithi188.stdout: "must": true, 2024-02-05T04:34:57.954 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "0.000000" 2024-02-05T04:34:57.954 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:34:57.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-02-05T04:35:00.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:35:01.223 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:35:01.223 INFO:tasks.workunit.client.0.smithi188.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-05T04:34:58.029968+0000 26'7 26:38 [1,2,0] 1 [1,2,0] 1 26'7 2024-02-05T04:34:58.029899+0000 26'7 2024-02-05T04:34:58.029899+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:34:58.029899+0000 4 0 2024-02-05T04:35:01.223 INFO:tasks.workunit.client.0.smithi188.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:34:48.573029+0000 0'0 26:25 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:01.223 INFO:tasks.workunit.client.0.smithi188.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:34:48.045317+0000 0'0 26:24 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:01.223 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:34:48.573039+0000 0'0 26:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:01.223 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:01.223 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:35:01.224 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:35:01.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-02-05T04:35:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:35:01.482 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:35:01.482 INFO:tasks.workunit.client.0.smithi188.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-05T04:34:58.029968+0000 26'7 26:38 [1,2,0] 1 [1,2,0] 1 26'7 2024-02-05T04:34:58.029899+0000 26'7 2024-02-05T04:34:58.029899+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:34:58.029899+0000 4 0 2024-02-05T04:35:01.482 INFO:tasks.workunit.client.0.smithi188.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:34:48.573029+0000 0'0 26:25 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:01.482 INFO:tasks.workunit.client.0.smithi188.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:34:48.045317+0000 0'0 26:24 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:01.482 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:34:48.573039+0000 0'0 26:24 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:01.482 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:01.482 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:35:01.483 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:35:01.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-02-05T04:35:01.876 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T04:35:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-02-05T04:35:02.245 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T04:35:02.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-02-05T04:35:07.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-02-05T04:35:07.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-02-05T04:35:07.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:35:07.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:35:07.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:35:07.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:35:07.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:35:07.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:35:07.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:07.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-05T04:35:07.593 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.592+0000 7feb5fc40800 1 bdev(0x556505a82000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.592+0000 7feb5fc40800 0 bdev(0x556505a82000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.593+0000 7feb5fc40800 1 bdev(0x556505a82000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.593+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.593+0000 7feb5fc40800 1 bdev(0x556505a82700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.593+0000 7feb5fc40800 0 bdev(0x556505a82700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.593+0000 7feb5fc40800 1 bdev(0x556505a82700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.593+0000 7feb5fc40800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:35:07.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.593+0000 7feb5fc40800 1 bluefs mount 2024-02-05T04:35:07.608 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.607+0000 7feb5fc40800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:07.633 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.633+0000 7feb5fc40800 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:35:07.633 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.633+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:07.636 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.635+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.635+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.635+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.635+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.635+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.635+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.635+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.636+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.636+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.636+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.636+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:07.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.636+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:07.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.669+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:07.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.669+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-02-05T04:35:07.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.669+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-02-05T04:35:07.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.669+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:35:07.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.669+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:35:07.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.669+0000 7feb5fc40800 1 freelist init 2024-02-05T04:35:07.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.669+0000 7feb5fc40800 1 freelist _read_cfg 2024-02-05T04:35:07.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.670+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.77383e-07 2024-02-05T04:35:07.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.670+0000 7feb5fc40800 1 bluefs umount 2024-02-05T04:35:07.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.670+0000 7feb5fc40800 1 bdev(0x556505a82700 td/osd-mapper/0/block) close 2024-02-05T04:35:07.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.855+0000 7feb5fc40800 1 bdev(0x556505a82700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:35:07.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.855+0000 7feb5fc40800 0 bdev(0x556505a82700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-05T04:35:07.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.855+0000 7feb5fc40800 1 bdev(0x556505a82700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:07.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.855+0000 7feb5fc40800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:35:07.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.855+0000 7feb5fc40800 1 bluefs mount 2024-02-05T04:35:07.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.855+0000 7feb5fc40800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:07.862 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.861+0000 7feb5fc40800 1 bluefs mount shared_bdev_used = 27459584 2024-02-05T04:35:07.862 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.861+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:07.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.863+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:07.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.863+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:07.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.863+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:07.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.863+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:07.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:07.864+0000 7feb5fc40800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:08.536 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.537+0000 7feb5fc40800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 0b 05 58 02 00 00 a6 98 61 bc 62 1a 4a 76 bd f0 |..X.....a.b.Jv..| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 d4 3d 01 56 f6 f2 00 00 00 00 1c 00 00 00 00 00 |.=.V............| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:* 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:35:08.539 INFO:tasks.workunit.client.0.smithi188.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:35:08.540 INFO:tasks.workunit.client.0.smithi188.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 61 6c ff a0 c8 ee |re........al....| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000230 44 5b 9b ff 87 92 13 3f 02 f8 00 00 00 00 00 00 |D[.....?........| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000240 00 00 1c 00 00 00 23 65 c0 65 ad db 35 2f 01 00 |......#e.e..5/..| 2024-02-05T04:35:08.541 INFO:tasks.workunit.client.0.smithi188.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:0000025e 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:08.542 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:0000003f 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:35:08.543 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:08.544 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:08.545 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:35:08.546 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:08.547 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.537+0000 7feb5fc40800 1 bluefs umount 2024-02-05T04:35:08.547 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.537+0000 7feb5fc40800 1 bdev(0x556505a82700 td/osd-mapper/0/block) close 2024-02-05T04:35:08.781 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.781+0000 7feb5fc40800 1 freelist shutdown 2024-02-05T04:35:08.781 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.781+0000 7feb5fc40800 1 bdev(0x556505a82000 td/osd-mapper/0/block) close 2024-02-05T04:35:08.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:08.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:35:08.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-02-05T04:35:08.884 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.883+0000 7f4642613800 1 bdev(0x5617647f6000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:35:08.884 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.883+0000 7f4642613800 0 bdev(0x5617647f6000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-05T04:35:08.884 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.883+0000 7f4642613800 1 bdev(0x5617647f6000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.883+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:35:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.884+0000 7f4642613800 1 bdev(0x5617647f6700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:35:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.884+0000 7f4642613800 0 bdev(0x5617647f6700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-05T04:35:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.884+0000 7f4642613800 1 bdev(0x5617647f6700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.884+0000 7f4642613800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:35:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.884+0000 7f4642613800 1 bluefs mount 2024-02-05T04:35:08.892 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.892+0000 7f4642613800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:08.900 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.900+0000 7f4642613800 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:35:08.900 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.900+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.902+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.902+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.902+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.903+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:08.936 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.935+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:08.936 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.935+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-02-05T04:35:08.936 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.935+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-02-05T04:35:08.936 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.935+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:35:08.936 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.935+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:35:08.936 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.935+0000 7f4642613800 1 freelist init 2024-02-05T04:35:08.936 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.935+0000 7f4642613800 1 freelist _read_cfg 2024-02-05T04:35:08.937 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.936+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-02-05T04:35:08.937 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.936+0000 7f4642613800 1 bluefs umount 2024-02-05T04:35:08.937 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:08.936+0000 7f4642613800 1 bdev(0x5617647f6700 td/osd-mapper/2/block) close 2024-02-05T04:35:09.146 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.146+0000 7f4642613800 1 bdev(0x5617647f6700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:35:09.147 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.146+0000 7f4642613800 0 bdev(0x5617647f6700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-05T04:35:09.147 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.146+0000 7f4642613800 1 bdev(0x5617647f6700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:09.147 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.146+0000 7f4642613800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:35:09.147 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.146+0000 7f4642613800 1 bluefs mount 2024-02-05T04:35:09.147 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.146+0000 7f4642613800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:09.153 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.152+0000 7f4642613800 1 bluefs mount shared_bdev_used = 27459584 2024-02-05T04:35:09.153 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.152+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.154+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.154+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.154+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:09.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:09.155+0000 7f4642613800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:10.075 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:10.074+0000 7f4642613800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:10.076 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:10.075+0000 7f4642613800 1 bluefs umount 2024-02-05T04:35:10.076 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:10.075+0000 7f4642613800 1 bdev(0x5617647f6700 td/osd-mapper/2/block) close 2024-02-05T04:35:10.315 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:10.315+0000 7f4642613800 1 freelist shutdown 2024-02-05T04:35:10.315 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:10.315+0000 7f4642613800 1 bdev(0x5617647f6000 td/osd-mapper/2/block) close 2024-02-05T04:35:10.400 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:10.400 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:10.400 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:10.401 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:10.401 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:10.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:10.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-02-05T04:35:10.402 INFO:tasks.workunit.client.0.smithi188.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-02-05T04:35:10.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:10.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:35:11.119 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 0b 05 58 02 00 00 a6 98 61 bc 62 1a 4a 76 bd f0 |..X.....a.b.Jv..| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 d4 3d 01 56 f6 f2 02 00 00 00 1c 00 00 00 00 00 |.=.V............| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:* 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:35:11.120 INFO:tasks.workunit.client.0.smithi188.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:35:11.121 INFO:tasks.workunit.client.0.smithi188.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 9e 81 8c a3 45 b3 |re............E.| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:00000230 47 2f 90 78 83 2e ac be 84 d1 00 00 00 00 00 00 |G/.x............| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:00000240 00 00 1c 00 00 00 65 65 c0 65 bd 1e f2 29 01 00 |......ee.e...)..| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-02-05T04:35:11.122 INFO:tasks.workunit.client.0.smithi188.stdout:0000025e 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:11.123 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:0000003f 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:11.124 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:11.125 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:11.127 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:11.127 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.127 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:35:11.127 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.127 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:35:11.128 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:11.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:35:11.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-02-05T04:35:11.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:35:11.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-02-05T04:35:12.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-02-05T04:35:12.741 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-02-05T04:35:12.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-02-05T04:35:12.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:12.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-02-05T04:35:12.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-02-05T04:35:12.744 INFO:tasks.workunit.client.0.smithi188.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-02-05T04:35:12.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-05T04:35:12.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 1 bdev(0x56419498c000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 0 bdev(0x56419498c000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 1 bdev(0x56419498c000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 1 bdev(0x56419498c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 0 bdev(0x56419498c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 1 bdev(0x56419498c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:35:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.769+0000 7fa50c00c800 1 bluefs mount 2024-02-05T04:35:12.790 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.789+0000 7fa50c00c800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:12.833 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.833+0000 7fa50c00c800 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:35:12.833 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.833+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:12.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:12.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:12.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:12.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:12.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:12.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:12.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:12.837 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:12.837 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:12.837 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:12.837 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:12.837 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.835+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:12.840 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.840+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:12.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.840+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-02-05T04:35:12.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.840+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-02-05T04:35:12.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.840+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:35:12.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.840+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:35:12.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.840+0000 7fa50c00c800 1 freelist init 2024-02-05T04:35:12.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.840+0000 7fa50c00c800 1 freelist _read_cfg 2024-02-05T04:35:12.842 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.841+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 8.77383e-07 2024-02-05T04:35:12.842 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.841+0000 7fa50c00c800 1 bluefs umount 2024-02-05T04:35:12.842 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:12.841+0000 7fa50c00c800 1 bdev(0x56419498c700 td/osd-mapper/0/block) close 2024-02-05T04:35:13.031 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.031+0000 7fa50c00c800 1 bdev(0x56419498c700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-02-05T04:35:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.031+0000 7fa50c00c800 0 bdev(0x56419498c700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-02-05T04:35:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.031+0000 7fa50c00c800 1 bdev(0x56419498c700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.031+0000 7fa50c00c800 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-02-05T04:35:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.031+0000 7fa50c00c800 1 bluefs mount 2024-02-05T04:35:13.040 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.039+0000 7fa50c00c800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:13.048 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.048+0000 7fa50c00c800 1 bluefs mount shared_bdev_used = 9568256 2024-02-05T04:35:13.048 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.048+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:13.051 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:13.051 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.050+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:13.052 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.051+0000 7fa50c00c800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:13.292 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.292+0000 7fa50c00c800 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 0b 05 58 02 00 00 a6 98 61 bc 62 1a 4a 76 bd f0 |..X.....a.b.Jv..| 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 d4 3d 01 56 f6 f2 00 00 00 00 1c 00 00 00 00 00 |.=.V............| 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:* 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:35:13.294 INFO:tasks.workunit.client.0.smithi188.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:35:13.295 INFO:tasks.workunit.client.0.smithi188.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000220 72 65 1c 00 00 00 05 00 00 00 61 6c ff a0 c8 ee |re........al....| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000230 44 5b 9b ff 87 92 13 3f 02 f8 00 00 00 00 00 00 |D[.....?........| 2024-02-05T04:35:13.296 INFO:tasks.workunit.client.0.smithi188.stdout:00000240 00 00 1c 00 00 00 23 65 c0 65 ad db 35 2f 01 00 |......#e.e..5/..| 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:0000025e 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:13.297 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:0000003f 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.298 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:13.299 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:13.300 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:13.301 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.292+0000 7fa50c00c800 1 bluefs umount 2024-02-05T04:35:13.302 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.292+0000 7fa50c00c800 1 bdev(0x56419498c700 td/osd-mapper/0/block) close 2024-02-05T04:35:13.529 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.529+0000 7fa50c00c800 1 freelist shutdown 2024-02-05T04:35:13.529 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:13.529+0000 7fa50c00c800 1 bdev(0x56419498c000 td/osd-mapper/0/block) close 2024-02-05T04:35:13.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-02-05T04:35:13.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-05T04:35:13.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:35:14.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:14.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:14.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-05T04:35:14.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-05T04:35:14.826 INFO:tasks.workunit.client.0.smithi188.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:14.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-05T04:35:14.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.Z9mrHhaPCA_the_val 2024-02-05T04:35:14.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:14.828 INFO:tasks.workunit.client.0.smithi188.stdout:Value dumped in: /tmp/tmp.Z9mrHhaPCA_the_val 2024-02-05T04:35:14.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.Z9mrHhaPCA_the_val' 2024-02-05T04:35:14.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.Z9mrHhaPCA_the_val 2024-02-05T04:35:15.345 INFO:tasks.workunit.client.0.smithi188.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-05T04:35:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.Z9mrHhaPCA_the_val 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-05T04:35:15.646 INFO:tasks.workunit.client.0.smithi188.stdout:0000073 2024-02-05T04:35:15.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:35:15.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:16.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.Z9mrHhaPCA_the_val 2024-02-05T04:35:17.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:17.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:17.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-02-05T04:35:17.734 INFO:tasks.workunit.client.0.smithi188.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-02-05T04:35:17.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-02-05T04:35:17.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-05T04:35:17.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:35:17.759 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.758+0000 7f292dc35800 1 bdev(0x5633ce7b0000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-05T04:35:17.759 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.758+0000 7f292dc35800 0 bdev(0x5633ce7b0000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-02-05T04:35:17.759 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.758+0000 7f292dc35800 1 bdev(0x5633ce7b0000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:17.760 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.758+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:35:17.760 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.758+0000 7f292dc35800 1 bdev(0x5633ce7b0700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-05T04:35:17.760 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.759+0000 7f292dc35800 0 bdev(0x5633ce7b0700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-02-05T04:35:17.760 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.759+0000 7f292dc35800 1 bdev(0x5633ce7b0700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:17.760 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.759+0000 7f292dc35800 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-02-05T04:35:17.760 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.759+0000 7f292dc35800 1 bluefs mount 2024-02-05T04:35:17.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.769+0000 7f292dc35800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:17.779 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.778+0000 7f292dc35800 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:35:17.779 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.778+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.780+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:17.782 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:17.783 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:17.783 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:17.783 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:17.783 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.781+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:17.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.786+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:17.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.786+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-02-05T04:35:17.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.786+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-02-05T04:35:17.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.786+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:35:17.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.786+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:35:17.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.786+0000 7f292dc35800 1 freelist init 2024-02-05T04:35:17.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.786+0000 7f292dc35800 1 freelist _read_cfg 2024-02-05T04:35:17.788 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.787+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 17 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 6.10353e-07 2024-02-05T04:35:17.788 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.787+0000 7f292dc35800 1 bluefs umount 2024-02-05T04:35:17.788 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:17.787+0000 7f292dc35800 1 bdev(0x5633ce7b0700 td/osd-mapper/1/block) close 2024-02-05T04:35:18.021 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.021+0000 7f292dc35800 1 bdev(0x5633ce7b0700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-02-05T04:35:18.022 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.021+0000 7f292dc35800 0 bdev(0x5633ce7b0700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-02-05T04:35:18.022 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.021+0000 7f292dc35800 1 bdev(0x5633ce7b0700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:18.022 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.021+0000 7f292dc35800 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-02-05T04:35:18.022 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.021+0000 7f292dc35800 1 bluefs mount 2024-02-05T04:35:18.036 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.036+0000 7f292dc35800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:18.044 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.044+0000 7f292dc35800 1 bluefs mount shared_bdev_used = 9568256 2024-02-05T04:35:18.045 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.044+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:18.048 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:18.048 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.046+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.047+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.047+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.047+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:18.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.047+0000 7f292dc35800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:18.300 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.300+0000 7f292dc35800 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 0b 05 58 02 00 00 a6 98 61 bc 62 1a 4a 76 bd f0 |..X.....a.b.Jv..| 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 d4 3d 01 56 f6 f2 01 00 00 00 1c 00 00 00 00 00 |.=.V............| 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:* 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:35:18.302 INFO:tasks.workunit.client.0.smithi188.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:35:18.303 INFO:tasks.workunit.client.0.smithi188.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000220 72 65 1c 00 00 00 0a 00 00 00 16 34 d1 c8 66 75 |re.........4..fu| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000230 40 25 94 47 c3 a8 61 49 ff 8a 00 00 00 00 00 00 |@%.G..aI........| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000240 00 00 1c 00 00 00 44 65 c0 65 88 18 a5 39 01 00 |......De.e...9..| 2024-02-05T04:35:18.304 INFO:tasks.workunit.client.0.smithi188.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:0000025e 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:35:18.305 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:0000003f 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:18.306 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:18.307 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:18.308 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.301+0000 7f292dc35800 1 bluefs umount 2024-02-05T04:35:18.309 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.301+0000 7f292dc35800 1 bdev(0x5633ce7b0700 td/osd-mapper/1/block) close 2024-02-05T04:35:18.537 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.537+0000 7f292dc35800 1 freelist shutdown 2024-02-05T04:35:18.537 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:18.537+0000 7f292dc35800 1 bdev(0x5633ce7b0000 td/osd-mapper/1/block) close 2024-02-05T04:35:18.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:35:18.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-05T04:35:18.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:35:19.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:19.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:19.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-05T04:35:19.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-05T04:35:19.574 INFO:tasks.workunit.client.0.smithi188.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:19.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-05T04:35:19.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.WOuceMuRml_the_val 2024-02-05T04:35:19.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:19.576 INFO:tasks.workunit.client.0.smithi188.stdout:Value dumped in: /tmp/tmp.WOuceMuRml_the_val 2024-02-05T04:35:19.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.WOuceMuRml_the_val' 2024-02-05T04:35:19.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.WOuceMuRml_the_val 2024-02-05T04:35:20.086 INFO:tasks.workunit.client.0.smithi188.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-05T04:35:20.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:20.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.WOuceMuRml_the_val 2024-02-05T04:35:20.369 INFO:tasks.workunit.client.0.smithi188.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-05T04:35:20.369 INFO:tasks.workunit.client.0.smithi188.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-05T04:35:20.369 INFO:tasks.workunit.client.0.smithi188.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-05T04:35:20.369 INFO:tasks.workunit.client.0.smithi188.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-05T04:35:20.369 INFO:tasks.workunit.client.0.smithi188.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-05T04:35:20.369 INFO:tasks.workunit.client.0.smithi188.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-02-05T04:35:20.370 INFO:tasks.workunit.client.0.smithi188.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-05T04:35:20.370 INFO:tasks.workunit.client.0.smithi188.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-05T04:35:20.370 INFO:tasks.workunit.client.0.smithi188.stdout:0000073 2024-02-05T04:35:20.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:35:20.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:21.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.WOuceMuRml_the_val 2024-02-05T04:35:22.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:22.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:22.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-02-05T04:35:22.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-02-05T04:35:22.458 INFO:tasks.workunit.client.0.smithi188.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-02-05T04:35:22.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-02-05T04:35:22.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:35:22.481 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 1 bdev(0x559c12c7c000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 0 bdev(0x559c12c7c000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 1 bdev(0x559c12c7c000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 1 bdev(0x559c12c7c700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 0 bdev(0x559c12c7c700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 1 bdev(0x559c12c7c700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:35:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.481+0000 7f10dd248800 1 bluefs mount 2024-02-05T04:35:22.489 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.488+0000 7f10dd248800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.498+0000 7f10dd248800 1 bluefs mount shared_bdev_used = 0 2024-02-05T04:35:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.498+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:22.501 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.500+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.500+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.500+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.500+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.500+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.501+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.501+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.501+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.501+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.501+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.501+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:22.502 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.501+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:22.506 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.505+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.506+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.506+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.506+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.506+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.506+0000 7f10dd248800 1 freelist init 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.506+0000 7f10dd248800 1 freelist _read_cfg 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.506+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc0000, fragmentation 4.95912e-07 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.507+0000 7f10dd248800 1 bluefs umount 2024-02-05T04:35:22.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.507+0000 7f10dd248800 1 bdev(0x559c12c7c700 td/osd-mapper/2/block) close 2024-02-05T04:35:22.744 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.743+0000 7f10dd248800 1 bdev(0x559c12c7c700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-02-05T04:35:22.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.743+0000 7f10dd248800 0 bdev(0x559c12c7c700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-02-05T04:35:22.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.743+0000 7f10dd248800 1 bdev(0x559c12c7c700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-02-05T04:35:22.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.743+0000 7f10dd248800 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-02-05T04:35:22.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.743+0000 7f10dd248800 1 bluefs mount 2024-02-05T04:35:22.867 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.755+0000 7f10dd248800 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-02-05T04:35:22.867 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.763+0000 7f10dd248800 1 bluefs mount shared_bdev_used = 9568256 2024-02-05T04:35:22.867 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.763+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-02-05T04:35:22.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:22.766+0000 7f10dd248800 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-02-05T04:35:23.008 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:23.007+0000 7f10dd248800 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 0b 05 58 02 00 00 a6 98 61 bc 62 1a 4a 76 bd f0 |..X.....a.b.Jv..| 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 d4 3d 01 56 f6 f2 02 00 00 00 1c 00 00 00 00 00 |.=.V............| 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:* 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-02-05T04:35:23.010 INFO:tasks.workunit.client.0.smithi188.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-02-05T04:35:23.011 INFO:tasks.workunit.client.0.smithi188.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000220 72 65 1c 00 00 00 0f 00 00 00 9e 81 8c a3 45 b3 |re............E.| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000230 47 2f 90 78 83 2e ac be 84 d1 00 00 00 00 00 00 |G/.x............| 2024-02-05T04:35:23.012 INFO:tasks.workunit.client.0.smithi188.stdout:00000240 00 00 1c 00 00 00 65 65 c0 65 bd 1e f2 29 01 00 |......ee.e...)..| 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1c 00 00 00 |..............| 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:0000025e 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:23.013 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:0000003f 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000047 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:23.014 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:23.015 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-02-05T04:35:23.016 INFO:tasks.workunit.client.0.smithi188.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout:0000003b 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:23.009+0000 7f10dd248800 1 bluefs umount 2024-02-05T04:35:23.017 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:23.009+0000 7f10dd248800 1 bdev(0x559c12c7c700 td/osd-mapper/2/block) close 2024-02-05T04:35:23.253 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:23.253+0000 7f10dd248800 1 freelist shutdown 2024-02-05T04:35:23.254 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:23.253+0000 7f10dd248800 1 bdev(0x559c12c7c000 td/osd-mapper/2/block) close 2024-02-05T04:35:23.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-02-05T04:35:23.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-02-05T04:35:23.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:35:24.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:24.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:24.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-02-05T04:35:24.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-02-05T04:35:24.301 INFO:tasks.workunit.client.0.smithi188.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:24.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-02-05T04:35:24.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.IQCSdrkPrW_the_val 2024-02-05T04:35:24.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:24.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.IQCSdrkPrW_the_val' 2024-02-05T04:35:24.303 INFO:tasks.workunit.client.0.smithi188.stdout:Value dumped in: /tmp/tmp.IQCSdrkPrW_the_val 2024-02-05T04:35:24.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.IQCSdrkPrW_the_val 2024-02-05T04:35:24.844 INFO:tasks.workunit.client.0.smithi188.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-02-05T04:35:25.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:25.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.IQCSdrkPrW_the_val 2024-02-05T04:35:25.096 INFO:tasks.workunit.client.0.smithi188.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-02-05T04:35:25.096 INFO:tasks.workunit.client.0.smithi188.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-02-05T04:35:25.096 INFO:tasks.workunit.client.0.smithi188.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-02-05T04:35:25.096 INFO:tasks.workunit.client.0.smithi188.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-02-05T04:35:25.096 INFO:tasks.workunit.client.0.smithi188.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-02-05T04:35:25.097 INFO:tasks.workunit.client.0.smithi188.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-02-05T04:35:25.097 INFO:tasks.workunit.client.0.smithi188.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-02-05T04:35:25.097 INFO:tasks.workunit.client.0.smithi188.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-02-05T04:35:25.097 INFO:tasks.workunit.client.0.smithi188.stdout:0000073 2024-02-05T04:35:25.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:35:25.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:26.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.IQCSdrkPrW_the_val 2024-02-05T04:35:27.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:27.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 ' 2024-02-05T04:35:27.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 2024-02-05T04:35:27.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:35:27.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:27.435 INFO:tasks.workunit.client.0.smithi188.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ 2024-02-05T04:35:27.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1/' 2024-02-05T04:35:27.435 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-02-05T04:35:27.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:35:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27892/$cluster-$name.asok' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27892/$cluster-$name.asok' 2024-02-05T04:35:27.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:35:27.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-02-05T04:35:27.443 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:35:27.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:35:27.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27892/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:35:27.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-02-05T04:35:27.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:35:27.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:35:27.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:35:27.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:35:27.477 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:27.476+0000 7fee696134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:27.479 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:27.479+0000 7fee696134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:27.480 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:27.480+0000 7fee696134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:27.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:35:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:35:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:35:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:35:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:35:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:27.726 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:35:27.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:35:27.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:27.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:35:27.995 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:27.995+0000 7fee696134c0 -1 Falling back to public interface 2024-02-05T04:35:27.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:29.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:29.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:29.001 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:35:29.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:35:29.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:29.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:35:29.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:30.262 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:30.262+0000 7fee696134c0 -1 osd.0 28 log_to_monitors true 2024-02-05T04:35:30.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:30.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:30.279 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:35:30.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:35:30.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:30.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:35:30.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:31.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:31.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:31.563 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:35:31.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:35:31.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:31.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:35:31.847 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 31 up_thru 31 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/2397930952,v1:127.0.0.1:6803/2397930952] [v2:127.0.0.1:6804/2397930952,v1:127.0.0.1:6805/2397930952] exists,up 616cffa0-c8ee-445b-9bff-8792133f02f8 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:35:31.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:35:31.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27892/$cluster-$name.asok' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27892/$cluster-$name.asok' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:35:31.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-02-05T04:35:31.854 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:35:31.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:35:31.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27892/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:35:31.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-02-05T04:35:31.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:35:31.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:35:31.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:35:31.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:35:31.883 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:31.883+0000 7fad1662b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:31.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:31.885+0000 7fad1662b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:31.886 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:31.886+0000 7fad1662b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:32.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:35:32.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:35:32.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:35:32.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:35:32.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:35:32.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:32.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:35:32.171 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:35:32.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:32.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:35:32.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:32.400+0000 7fad1662b4c0 -1 Falling back to public interface 2024-02-05T04:35:32.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:33.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:33.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:33.448 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:35:33.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:35:33.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:33.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:35:33.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:34.723 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:34.723+0000 7fad1662b4c0 -1 osd.1 28 log_to_monitors true 2024-02-05T04:35:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:34.736 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:35:34.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:35:34.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:34.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:35:35.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:36.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:36.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:36.018 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:35:36.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:35:36.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:36.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:35:36.315 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3596031812,v1:127.0.0.1:6811/3596031812] [v2:127.0.0.1:6812/3596031812,v1:127.0.0.1:6813/3596031812] exists,up 1634d1c8-6675-4025-9447-c3a86149ff8a 2024-02-05T04:35:36.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:35:36.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:35:36.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:35:36.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:36.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:35:36.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-02-05T04:35:36.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1' 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:35:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27892/$cluster-$name.asok' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27892/$cluster-$name.asok' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:35:36.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:35:36.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:35:36.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:35:36.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-02-05T04:35:36.322 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:35:36.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T04:35:36.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=a69861bc-621a-4a76-bdf0-d43d0156f6f2 --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.27892/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:35:36.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-02-05T04:35:36.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T04:35:36.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:35:36.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:35:36.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:35:36.351 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:36.351+0000 7f1a903fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:36.353 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:36.353+0000 7f1a903fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:36.354 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:36.354+0000 7f1a903fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:35:36.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T04:35:36.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:35:36.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:35:36.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:35:36.635 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:35:36.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:35:36.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:36.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:35:36.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:36.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:35:36.867 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:36.867+0000 7f1a903fb4c0 -1 Falling back to public interface 2024-02-05T04:35:36.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:37.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:37.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:37.915 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:35:37.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:35:37.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:37.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:35:38.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:39.161 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:35:39.160+0000 7f1a903fb4c0 -1 osd.2 28 log_to_monitors true 2024-02-05T04:35:39.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:39.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:39.196 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:35:39.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:35:39.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:39.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:35:39.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:35:40.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:35:40.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:35:40.529 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:35:40.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:35:40.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:35:40.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:35:40.977 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 38 up_thru 38 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/3925862159,v1:127.0.0.1:6819/3925862159] [v2:127.0.0.1:6820/3925862159,v1:127.0.0.1:6821/3925862159] exists,up 9e818ca3-45b3-472f-9078-832eacbe84d1 2024-02-05T04:35:40.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:35:40.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:35:40.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:35:40.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-02-05T04:35:41.979 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-02-05T04:35:41.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-02-05T04:35:41.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:41.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-02-05T04:35:42.086 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:35:42.086 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "19.0.0-432-g3cfd0d29", 2024-02-05T04:35:42.086 INFO:tasks.workunit.client.0.smithi188.stdout: "release": "squid", 2024-02-05T04:35:42.086 INFO:tasks.workunit.client.0.smithi188.stdout: "release_type": "dev" 2024-02-05T04:35:42.086 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:35:42.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:42.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-02-05T04:35:42.190 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:35:42.190 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "19.0.0-432-g3cfd0d29", 2024-02-05T04:35:42.191 INFO:tasks.workunit.client.0.smithi188.stdout: "release": "squid", 2024-02-05T04:35:42.191 INFO:tasks.workunit.client.0.smithi188.stdout: "release_type": "dev" 2024-02-05T04:35:42.191 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:35:42.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-02-05T04:35:42.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-02-05T04:35:42.294 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:35:42.294 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "19.0.0-432-g3cfd0d29", 2024-02-05T04:35:42.294 INFO:tasks.workunit.client.0.smithi188.stdout: "release": "squid", 2024-02-05T04:35:42.294 INFO:tasks.workunit.client.0.smithi188.stdout: "release_type": "dev" 2024-02-05T04:35:42.295 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:35:42.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-02-05T04:35:42.330 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:35:42.330 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "objxxx", 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 2, 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [ 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 1, 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap01" 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.331 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 2, 2024-02-05T04:35:42.332 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap02" 2024-02-05T04:35:42.332 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:35:42.332 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T04:35:42.332 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 6, 2024-02-05T04:35:42.332 INFO:tasks.workunit.client.0.smithi188.stdout: "overlaps": [] 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 3, 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [ 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 3, 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap13" 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:35:42.333 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T04:35:42.334 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 5, 2024-02-05T04:35:42.334 INFO:tasks.workunit.client.0.smithi188.stdout: "overlaps": [] 2024-02-05T04:35:42.334 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:35:42.334 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.334 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 5, 2024-02-05T04:35:42.334 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [ 2024-02-05T04:35:42.335 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.335 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 4, 2024-02-05T04:35:42.335 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap24" 2024-02-05T04:35:42.335 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:35:42.335 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.335 INFO:tasks.workunit.client.0.smithi188.stdout: "id": 5, 2024-02-05T04:35:42.336 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "snap25" 2024-02-05T04:35:42.336 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:35:42.336 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T04:35:42.336 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 6, 2024-02-05T04:35:42.336 INFO:tasks.workunit.client.0.smithi188.stdout: "overlaps": [] 2024-02-05T04:35:42.336 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T04:35:42.337 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T04:35:42.337 INFO:tasks.workunit.client.0.smithi188.stdout: "id": "head", 2024-02-05T04:35:42.337 INFO:tasks.workunit.client.0.smithi188.stdout: "snapshots": [], 2024-02-05T04:35:42.337 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 6 2024-02-05T04:35:42.337 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T04:35:42.337 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T04:35:42.338 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:35:42.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-02-05T04:35:42.693 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is unset 2024-02-05T04:35:42.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-02-05T04:35:43.195 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is unset 2024-02-05T04:35:43.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-02-05T04:35:43.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-02-05T04:35:43.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-02-05T04:35:43.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:35:43.727 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:35:43.727 INFO:tasks.workunit.client.0.smithi188.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 peering 2024-02-05T04:35:39.868165+0000 26'7 37:47 [1,2,0] 1 [1,2,0] 1 26'7 2024-02-05T04:34:58.029899+0000 26'7 2024-02-05T04:34:58.029899+0000 0 1 no scrub is scheduled 4 0 2024-02-05T04:35:43.727 INFO:tasks.workunit.client.0.smithi188.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:35:40.371246+0000 0'0 39:37 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:43.728 INFO:tasks.workunit.client.0.smithi188.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2024-02-05T04:35:35.419487+0000 0'0 34:10 [0,1] 0 [0,1] 0 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 no scrub is scheduled 0 0 2024-02-05T04:35:43.728 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-02-05T04:35:39.868241+0000 0'0 37:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 no scrub is scheduled 0 0 2024-02-05T04:35:43.728 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:43.728 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:35:43.728 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:35:43.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-02-05T04:35:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-05T04:35:45.844 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:35:45.845 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T04:35:45.845 INFO:tasks.workunit.client.0.smithi188.stdout: "must": true, 2024-02-05T04:35:45.845 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "0.000000" 2024-02-05T04:35:45.845 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:35:45.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-02-05T04:35:50.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:35:51.116 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:35:51.116 INFO:tasks.workunit.client.0.smithi188.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-05T04:35:46.719473+0000 26'7 42:69 [1,2,0] 1 [1,2,0] 1 26'7 2024-02-05T04:35:46.719395+0000 26'7 2024-02-05T04:35:46.719395+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:35:46.719395+0000 4 0 2024-02-05T04:35:51.116 INFO:tasks.workunit.client.0.smithi188.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:35:40.371246+0000 0'0 42:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:51.116 INFO:tasks.workunit.client.0.smithi188.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:35:40.371380+0000 0'0 41:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:51.116 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:35:40.715253+0000 0'0 41:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:51.117 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:51.117 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:35:51.117 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:35:51.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:51.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:35:51.128 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:35:46.718+0000 7facf8401640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-02-05T04:35:51.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-02-05T04:35:51.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:35:51.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-02-05T04:35:51.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-02-05T04:35:51.133 INFO:tasks.workunit.client.0.smithi188.stdout:prev count: 1 2024-02-05T04:35:51.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-02-05T04:35:51.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-02-05T04:35:51.225 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:35:51.225 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T04:35:51.225 INFO:tasks.workunit.client.0.smithi188.stdout: "must": true, 2024-02-05T04:35:51.226 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "0.000000" 2024-02-05T04:35:51.226 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:35:51.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-02-05T04:35:56.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-02-05T04:35:56.494 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:35:56.494 INFO:tasks.workunit.client.0.smithi188.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-02-05T04:35:46.719473+0000 26'7 42:69 [1,2,0] 1 [1,2,0] 1 26'7 2024-02-05T04:35:46.719395+0000 26'7 2024-02-05T04:35:46.719395+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:35:46.719395+0000 4 0 2024-02-05T04:35:56.495 INFO:tasks.workunit.client.0.smithi188.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:35:40.371246+0000 0'0 42:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:56.495 INFO:tasks.workunit.client.0.smithi188.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:35:40.371380+0000 0'0 41:25 [2,0,1] 2 [2,0,1] 2 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:56.495 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:35:40.715253+0000 0'0 41:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:34:47.245742+0000 0'0 2024-02-05T04:34:47.245742+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:34:47.245742+0000 0 0 2024-02-05T04:35:56.495 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:35:56.495 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:35:56.495 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:35:56.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:56.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:35:56.506 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:35:46.718+0000 7facf8401640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-02-05T04:35:56.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-02-05T04:35:56.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-02-05T04:35:56.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-02-05T04:35:56.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-02-05T04:35:56.509 INFO:tasks.workunit.client.0.smithi188.stdout:current count: 1 2024-02-05T04:35:56.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-02-05T04:35:56.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-02-05T04:35:56.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-02-05T04:35:56.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:35:56.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:35:56.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:35:56.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:35:56.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:35:56.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:35:56.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-02-05T04:35:56.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-02-05T04:35:56.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:35:56.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-02-05T04:35:56.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:35:58.352 INFO:tasks.workunit.client.0.smithi188.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:58.352 INFO:tasks.workunit.client.0.smithi188.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-02-05T04:35:58.352 INFO:tasks.workunit.client.0.smithi188.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-02-05T04:35:58.353 INFO:tasks.workunit.client.0.smithi188.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-02-05T04:35:58.353 INFO:tasks.workunit.client.0.smithi188.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:58.353 INFO:tasks.workunit.client.0.smithi188.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-02-05T04:35:58.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-02-05T04:35:58.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-02-05T04:35:58.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-02-05T04:35:58.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-02-05T04:35:59.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-02-05T04:35:59.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-02-05T04:35:59.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-02-05T04:35:59.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-02-05T04:35:59.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-05T04:35:59.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:35:59.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-05T04:35:59.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:35:59.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:35:59.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:35:59.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:35:59.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:35:59.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:35:59.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:35:59.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:35:59.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:35:59.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:35:59.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:35:59.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:35:59.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:35:59.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:35:59.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:35:59.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:35:59.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:35:59.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:35:59.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-05T04:35:59.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:35:59.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:35:59.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:35:59.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27892 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-02-05T04:35:59.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:35:59.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:35:59.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:35:59.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:35:59.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:35:59.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:35:59.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:35:59.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:35:59.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:35:59.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:35:59.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:35:59.532 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:35:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:35:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:35:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:35:59.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:35:59.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:35:59.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T04:35:59.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-02-05T04:35:59.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:35:59.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:35:59.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27892 2024-02-05T04:35:59.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27892 2024-02-05T04:35:59.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:35:59.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:35:59.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T04:35:59.539 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T04:35:59.539 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T04:35:59.610 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-02-05T04:35:59.610 DEBUG:teuthology.orchestra.run.smithi188:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-02-05T04:35:59.673 INFO:tasks.workunit.client.0.smithi188.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T04:35:59.679 INFO:tasks.workunit.client.0.smithi188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T04:35:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none ' 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-02-05T04:35:59.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_recovery_scrub_2' 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.44176 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-02-05T04:35:59.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:35:59.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:35:59.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:35:59.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:35:59.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:35:59.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:35:59.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:35:59.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:35:59.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:35:59.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:35:59.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:35:59.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:35:59.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:35:59.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T04:35:59.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:35:59.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:35:59.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:35:59.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44176 2024-02-05T04:35:59.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:35:59.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:35:59.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-02-05T04:35:59.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:35:59.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:35:59.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:35:59.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44176 2024-02-05T04:35:59.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:35:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T04:35:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T04:35:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-02-05T04:35:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:36:00.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:36:00.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:36:00.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:36:00.056 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:36:00.056 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:00.056 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:00.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:00.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:36:00.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:36:00.250 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:36:00.251 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:36:00.251 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:00.251 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:00.251 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44176/ceph-mon.a.asok 2024-02-05T04:36:00.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:36:00.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44176/ceph-mon.a.asok config get fsid 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:36:00.334 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:00.335 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:00.335 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44176/ceph-mon.a.asok 2024-02-05T04:36:00.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:36:00.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44176/ceph-mon.a.asok config get mon_host 2024-02-05T04:36:00.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-02-05T04:36:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-02-05T04:36:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:36:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:36:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:36:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-02-05T04:36:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:36:01.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:36:01.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:36:01.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-02-05T04:36:01.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-02-05T04:36:01.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-02-05T04:36:01.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-02-05T04:36:01.254 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-02-05T04:36:01.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-02-05T04:36:01.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:36:01.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-02-05T04:36:01.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:36:01.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:36:01.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:36:01.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:36:01.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:36:01.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:36:01.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-02-05T04:36:01.262 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:36:01.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d48d583c-374d-4416-843d-e7d6cf04aea1 2024-02-05T04:36:01.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d48d583c-374d-4416-843d-e7d6cf04aea1' 2024-02-05T04:36:01.262 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 d48d583c-374d-4416-843d-e7d6cf04aea1 2024-02-05T04:36:01.262 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:36:01.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxZcBlP751EBAAenWzHkZ4nZO+cO/ryFGWdQ== 2024-02-05T04:36:01.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxZcBlP751EBAAenWzHkZ4nZO+cO/ryFGWdQ=="}' 2024-02-05T04:36:01.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d48d583c-374d-4416-843d-e7d6cf04aea1 -i td/osd-recovery-scrub/0/new.json 2024-02-05T04:36:01.482 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:36:01.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-02-05T04:36:01.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQCxZcBlP751EBAAenWzHkZ4nZO+cO/ryFGWdQ== --osd-uuid d48d583c-374d-4416-843d-e7d6cf04aea1 2024-02-05T04:36:01.521 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:01.520+0000 7f1522ad64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:01.533 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:01.533+0000 7f1522ad64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:01.534 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:01.534+0000 7f1522ad64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:01.535 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:01.534+0000 7f1522ad64c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:36:01.535 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:01.534+0000 7f1522ad64c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:36:01.535 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:01.534+0000 7f1522ad64c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:36:01.608 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:01.608+0000 7f1522ad64c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-02-05T04:36:06.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-02-05T04:36:06.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:36:06.898 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:36:06.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:36:06.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:36:07.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:36:07.247 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:36:07.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:36:07.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:36:07.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:36:07.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:36:07.273 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:07.272+0000 7f0e6c88a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:07.293 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:07.293+0000 7f0e6c88a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:07.294 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:07.294+0000 7f0e6c88a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:07.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:36:07.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:36:07.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:36:07.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:36:07.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:36:07.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:07.552 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:36:07.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:36:07.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:07.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:36:07.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:07.828 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:07.828+0000 7f0e6c88a4c0 -1 Falling back to public interface 2024-02-05T04:36:08.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:08.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:08.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:36:08.818 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:36:08.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:08.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:36:09.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:10.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:10.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:10.098 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:36:10.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:36:10.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:10.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:36:10.160 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:10.159+0000 7f0e6c88a4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:36:10.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:11.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:36:11.420 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:36:11.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:11.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:36:11.689 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1976084194,v1:127.0.0.1:6803/1976084194] [v2:127.0.0.1:6804/1976084194,v1:127.0.0.1:6805/1976084194] exists,up d48d583c-374d-4416-843d-e7d6cf04aea1 2024-02-05T04:36:11.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:36:11.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:36:11.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=22b52e8c-6135-4fef-81de-74a6a9f9f30e 2024-02-05T04:36:11.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 22b52e8c-6135-4fef-81de-74a6a9f9f30e' 2024-02-05T04:36:11.697 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 22b52e8c-6135-4fef-81de-74a6a9f9f30e 2024-02-05T04:36:11.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:36:11.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7ZcBl7uVNKhAADlHdi7stP9MO0l97WW8OqQ== 2024-02-05T04:36:11.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7ZcBl7uVNKhAADlHdi7stP9MO0l97WW8OqQ=="}' 2024-02-05T04:36:11.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 22b52e8c-6135-4fef-81de-74a6a9f9f30e -i td/osd-recovery-scrub/1/new.json 2024-02-05T04:36:12.048 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:36:12.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-02-05T04:36:12.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQC7ZcBl7uVNKhAADlHdi7stP9MO0l97WW8OqQ== --osd-uuid 22b52e8c-6135-4fef-81de-74a6a9f9f30e 2024-02-05T04:36:12.088 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:12.087+0000 7f562941a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:12.132 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:12.132+0000 7f562941a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:12.133 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:12.133+0000 7f562941a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:12.133 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:12.133+0000 7f562941a4c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:36:12.133 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:12.133+0000 7f562941a4c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:36:12.134 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:12.133+0000 7f562941a4c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:36:12.192 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:12.192+0000 7f562941a4c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-02-05T04:36:17.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-02-05T04:36:17.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:36:17.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:36:17.343 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:36:17.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:36:17.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:36:17.689 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:36:17.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:36:17.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:36:17.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:36:17.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:36:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:17.714+0000 7fa78892b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:17.717 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:17.716+0000 7fa78892b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:17.718 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:17.717+0000 7fa78892b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:17.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:36:17.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:36:17.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:36:17.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:36:17.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:36:17.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:17.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:36:17.960 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:36:17.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:17.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:36:18.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:18.251 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:18.251+0000 7fa78892b4c0 -1 Falling back to public interface 2024-02-05T04:36:19.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:19.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:19.226 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:36:19.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:36:19.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:19.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:36:19.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:20.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:20.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:20.525 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:36:20.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:36:20.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:20.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:36:20.674 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:20.673+0000 7fa78892b4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:36:20.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:21.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:21.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:21.892 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:36:21.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:36:21.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:21.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:36:22.169 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2702784013,v1:127.0.0.1:6811/2702784013] [v2:127.0.0.1:6812/2702784013,v1:127.0.0.1:6813/2702784013] exists,up 22b52e8c-6135-4fef-81de-74a6a9f9f30e 2024-02-05T04:36:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:36:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:36:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:36:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:36:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:36:22.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:36:22.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a07b39d-024a-40f2-a014-6bfd80807250 2024-02-05T04:36:22.176 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 6a07b39d-024a-40f2-a014-6bfd80807250 2024-02-05T04:36:22.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6a07b39d-024a-40f2-a014-6bfd80807250' 2024-02-05T04:36:22.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:36:22.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGZcBlzN9WCxAAG46jjVR2sWQf7cSItjYw2g== 2024-02-05T04:36:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGZcBlzN9WCxAAG46jjVR2sWQf7cSItjYw2g=="}' 2024-02-05T04:36:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a07b39d-024a-40f2-a014-6bfd80807250 -i td/osd-recovery-scrub/2/new.json 2024-02-05T04:36:22.537 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:36:22.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-02-05T04:36:22.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDGZcBlzN9WCxAAG46jjVR2sWQf7cSItjYw2g== --osd-uuid 6a07b39d-024a-40f2-a014-6bfd80807250 2024-02-05T04:36:22.577 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:22.576+0000 7f2e7a6634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:22.609 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:22.604+0000 7f2e7a6634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:22.609 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:22.605+0000 7f2e7a6634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:22.609 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:22.605+0000 7f2e7a6634c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:36:22.609 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:22.605+0000 7f2e7a6634c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:36:22.609 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:22.605+0000 7f2e7a6634c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:36:22.688 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:22.688+0000 7f2e7a6634c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-02-05T04:36:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-02-05T04:36:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:36:28.471 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:36:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:36:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:36:28.816 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:36:28.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:36:28.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:36:28.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:36:28.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:36:28.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:36:28.842 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:28.842+0000 7fed3440e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:28.844 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:28.844+0000 7fed3440e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:28.845 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:28.845+0000 7fed3440e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:29.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:36:29.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:36:29.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:36:29.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:36:29.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:36:29.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:29.088 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:36:29.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:36:29.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:29.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:36:29.368 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:29.367+0000 7fed3440e4c0 -1 Falling back to public interface 2024-02-05T04:36:29.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:30.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:30.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:30.377 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:36:30.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:36:30.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:30.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:36:30.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:31.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:31.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:31.651 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:36:31.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:36:31.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:31.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:36:31.705 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:31.705+0000 7fed3440e4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:36:31.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:32.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:32.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:32.923 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:36:32.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:36:32.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:32.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:36:33.237 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3042135958,v1:127.0.0.1:6819/3042135958] [v2:127.0.0.1:6820/3042135958,v1:127.0.0.1:6821/3042135958] exists,up 6a07b39d-024a-40f2-a014-6bfd80807250 2024-02-05T04:36:33.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:36:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:36:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:36:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c728f33-2549-451a-aa52-321776af32f0 2024-02-05T04:36:33.245 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 0c728f33-2549-451a-aa52-321776af32f0 2024-02-05T04:36:33.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0c728f33-2549-451a-aa52-321776af32f0' 2024-02-05T04:36:33.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:36:33.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRZcBlnTtuDxAA1b3wotvWPP/DWe5wfJ7JNw== 2024-02-05T04:36:33.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRZcBlnTtuDxAA1b3wotvWPP/DWe5wfJ7JNw=="}' 2024-02-05T04:36:33.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c728f33-2549-451a-aa52-321776af32f0 -i td/osd-recovery-scrub/3/new.json 2024-02-05T04:36:33.585 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:36:33.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-02-05T04:36:33.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDRZcBlnTtuDxAA1b3wotvWPP/DWe5wfJ7JNw== --osd-uuid 0c728f33-2549-451a-aa52-321776af32f0 2024-02-05T04:36:33.625 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:33.625+0000 7f5ad39704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:33.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:33.669+0000 7f5ad39704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:33.672 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:33.670+0000 7f5ad39704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:33.672 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:33.671+0000 7f5ad39704c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:36:33.672 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:33.671+0000 7f5ad39704c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:36:33.672 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:33.671+0000 7f5ad39704c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:36:33.736 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:33.736+0000 7f5ad39704c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-02-05T04:36:39.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-02-05T04:36:39.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:36:39.445 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T04:36:39.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T04:36:39.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:36:39.791 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T04:36:39.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T04:36:39.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-02-05T04:36:39.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:36:39.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:36:39.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:36:39.817 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:39.816+0000 7f3d48c074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:39.819 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:39.818+0000 7f3d48c074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:39.820 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:39.819+0000 7f3d48c074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:36:40.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T04:36:40.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:36:40.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T04:36:40.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:36:40.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:36:40.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:40.057 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:36:40.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:36:40.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:40.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:36:40.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:40.349 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:40.349+0000 7f3d48c074c0 -1 Falling back to public interface 2024-02-05T04:36:41.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:41.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:41.323 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:36:41.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:36:41.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:41.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:36:41.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:42.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:42.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:42.593 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:36:42.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:36:42.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:42.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:36:42.596 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:36:42.596+0000 7f3d48c074c0 -1 osd.3 0 log_to_monitors true 2024-02-05T04:36:42.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:36:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:36:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:36:43.906 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:36:43.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:36:43.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:36:43.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:36:44.183 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2773081761,v1:127.0.0.1:6827/2773081761] [v2:127.0.0.1:6828/2773081761,v1:127.0.0.1:6829/2773081761] exists,up 0c728f33-2549-451a-aa52-321776af32f0 2024-02-05T04:36:44.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:36:44.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:36:44.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:36:44.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-02-05T04:36:44.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T04:36:44.538 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T04:36:44.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:36:45.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-02-05T04:36:45.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:36:45.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:36:45.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:36:45.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:36:45.552 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:36:45.552 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:36:45.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:36:45.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:36:45.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:36:45.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:36:45.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:36:45.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:36:45.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:36:45.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:36:45.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:36:45.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:36:45.907 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:36:45.907 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:36:45.907 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T04:36:45.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:36:45.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:36:45.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:36:46.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-02-05T04:36:46.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-02-05T04:36:46.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-02-05T04:36:46.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:36:46.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:36:46.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-02-05T04:36:46.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-02-05T04:36:46.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672966' 2024-02-05T04:36:46.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:36:46.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:36:46.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-02-05T04:36:46.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-02-05T04:36:46.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444' 2024-02-05T04:36:46.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:36:46.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T04:36:46.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-02-05T04:36:46.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-02-05T04:36:46.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488 1-42949672966 2-64424509444 3-85899345922' 2024-02-05T04:36:46.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:36:46.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-02-05T04:36:46.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:36:46.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:36:46.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-02-05T04:36:46.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:36:46.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-02-05T04:36:46.321 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836488 2024-02-05T04:36:46.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-02-05T04:36:46.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:36:46.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836488 2024-02-05T04:36:46.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:36:47.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:36:47.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:36:47.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836488 2024-02-05T04:36:47.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:36:47.857 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-02-05T04:36:47.857 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:36:47.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:36:47.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-02-05T04:36:47.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:36:47.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-02-05T04:36:47.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-02-05T04:36:47.860 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672966 2024-02-05T04:36:47.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:36:48.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-02-05T04:36:48.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:36:48.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-02-05T04:36:48.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:36:48.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:36:48.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-02-05T04:36:48.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:36:48.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-02-05T04:36:48.132 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509444 2024-02-05T04:36:48.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-02-05T04:36:48.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:36:48.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-02-05T04:36:48.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:36:48.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-02-05T04:36:48.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:36:48.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T04:36:48.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-02-05T04:36:48.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:36:48.401 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345922 2024-02-05T04:36:48.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-02-05T04:36:48.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-02-05T04:36:48.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T04:36:48.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-02-05T04:36:48.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:36:48.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:36:48.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:36:48.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:36:48.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:36:48.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:36:48.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:36:48.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:36:48.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:36:48.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:36:48.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:36:49.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:36:49.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:36:49.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:36:49.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:36:49.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:36:49.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:36:49.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:36:49.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-02-05T04:36:49.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-02-05T04:36:49.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-02-05T04:36:49.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-02-05T04:36:49.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-02-05T04:36:50.118 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:36:50.118 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:36:45.350502+0000 0'0 23:8 [1] 1 [1] 1 0'0 2024-02-05T04:36:44.497716+0000 0'0 2024-02-05T04:36:44.497716+0000 0 0 periodic scrub scheduled @ 2024-02-06T14:23:02.049957+0000 0 0 2024-02-05T04:36:50.118 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:36:50.118 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:36:50.118 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:36:50.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.44176 bs=1M count=50 2024-02-05T04:36:50.266 INFO:tasks.workunit.client.0.smithi188.stderr:50+0 records in 2024-02-05T04:36:50.266 INFO:tasks.workunit.client.0.smithi188.stderr:50+0 records out 2024-02-05T04:36:50.266 INFO:tasks.workunit.client.0.smithi188.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.137975 s, 380 MB/s 2024-02-05T04:36:50.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-02-05T04:36:50.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:36:50.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.44176 2024-02-05T04:36:51.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:36:51.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.44176 2024-02-05T04:36:53.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:36:53.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.44176 2024-02-05T04:36:54.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:36:54.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.44176 2024-02-05T04:36:56.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:36:56.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.44176 2024-02-05T04:36:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:36:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.44176 2024-02-05T04:36:58.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:36:58.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.44176 2024-02-05T04:37:00.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:00.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.44176 2024-02-05T04:37:01.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:01.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.44176 2024-02-05T04:37:03.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:03.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.44176 2024-02-05T04:37:04.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:04.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.44176 2024-02-05T04:37:06.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:06.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.44176 2024-02-05T04:37:07.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:07.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.44176 2024-02-05T04:37:09.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:09.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.44176 2024-02-05T04:37:10.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:10.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.44176 2024-02-05T04:37:12.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:12.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.44176 2024-02-05T04:37:13.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:13.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.44176 2024-02-05T04:37:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.44176 2024-02-05T04:37:16.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:16.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.44176 2024-02-05T04:37:18.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:18.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.44176 2024-02-05T04:37:19.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:19.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.44176 2024-02-05T04:37:21.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:21.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.44176 2024-02-05T04:37:22.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:22.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.44176 2024-02-05T04:37:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.44176 2024-02-05T04:37:26.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:26.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.44176 2024-02-05T04:37:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.44176 2024-02-05T04:37:28.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:28.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.44176 2024-02-05T04:37:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.44176 2024-02-05T04:37:31.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:31.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.44176 2024-02-05T04:37:33.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:33.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.44176 2024-02-05T04:37:34.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:34.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.44176 2024-02-05T04:37:36.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:36.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.44176 2024-02-05T04:37:37.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:37.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.44176 2024-02-05T04:37:38.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:38.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.44176 2024-02-05T04:37:40.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:40.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.44176 2024-02-05T04:37:42.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:42.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.44176 2024-02-05T04:37:43.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:43.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.44176 2024-02-05T04:37:44.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:44.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.44176 2024-02-05T04:37:46.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:46.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.44176 2024-02-05T04:37:48.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:48.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.44176 2024-02-05T04:37:49.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:49.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.44176 2024-02-05T04:37:50.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:50.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.44176 2024-02-05T04:37:52.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:52.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.44176 2024-02-05T04:37:53.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:53.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.44176 2024-02-05T04:37:55.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:55.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.44176 2024-02-05T04:37:56.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:56.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.44176 2024-02-05T04:37:58.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:58.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.44176 2024-02-05T04:37:59.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:37:59.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.44176 2024-02-05T04:38:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.44176 2024-02-05T04:38:02.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:02.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.44176 2024-02-05T04:38:04.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:04.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.44176 2024-02-05T04:38:05.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:05.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.44176 2024-02-05T04:38:07.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:07.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.44176 2024-02-05T04:38:08.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:08.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.44176 2024-02-05T04:38:10.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:10.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.44176 2024-02-05T04:38:11.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:11.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.44176 2024-02-05T04:38:13.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:13.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.44176 2024-02-05T04:38:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.44176 2024-02-05T04:38:16.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:16.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.44176 2024-02-05T04:38:17.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:17.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.44176 2024-02-05T04:38:19.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:19.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.44176 2024-02-05T04:38:20.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:20.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.44176 2024-02-05T04:38:22.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:22.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.44176 2024-02-05T04:38:23.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:23.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.44176 2024-02-05T04:38:24.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:24.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.44176 2024-02-05T04:38:26.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:26.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.44176 2024-02-05T04:38:27.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:27.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.44176 2024-02-05T04:38:29.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:29.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.44176 2024-02-05T04:38:30.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:30.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.44176 2024-02-05T04:38:32.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:32.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.44176 2024-02-05T04:38:33.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:33.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.44176 2024-02-05T04:38:35.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:35.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.44176 2024-02-05T04:38:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.44176 2024-02-05T04:38:38.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:38.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.44176 2024-02-05T04:38:39.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:39.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.44176 2024-02-05T04:38:41.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:41.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.44176 2024-02-05T04:38:42.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:42.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.44176 2024-02-05T04:38:44.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:44.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.44176 2024-02-05T04:38:45.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:45.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.44176 2024-02-05T04:38:47.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:47.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.44176 2024-02-05T04:38:48.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:48.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.44176 2024-02-05T04:38:50.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:50.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.44176 2024-02-05T04:38:51.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:51.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.44176 2024-02-05T04:38:53.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:53.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.44176 2024-02-05T04:38:54.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:54.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.44176 2024-02-05T04:38:56.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:56.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.44176 2024-02-05T04:38:58.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:58.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.44176 2024-02-05T04:38:59.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:38:59.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.44176 2024-02-05T04:39:00.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:00.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.44176 2024-02-05T04:39:02.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:02.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.44176 2024-02-05T04:39:04.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:04.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.44176 2024-02-05T04:39:05.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:05.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.44176 2024-02-05T04:39:06.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:06.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.44176 2024-02-05T04:39:08.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:08.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.44176 2024-02-05T04:39:10.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:10.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.44176 2024-02-05T04:39:11.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:11.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.44176 2024-02-05T04:39:13.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:13.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.44176 2024-02-05T04:39:14.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:14.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.44176 2024-02-05T04:39:16.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:16.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.44176 2024-02-05T04:39:17.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-02-05T04:39:17.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.44176 2024-02-05T04:39:19.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.44176 2024-02-05T04:39:19.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-02-05T04:39:19.646 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 4 2024-02-05T04:39:19.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-02-05T04:39:19.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-02-05T04:39:19.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T04:39:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T04:39:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T04:39:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T04:39:19.909 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:39:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T04:39:21.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-02-05T04:39:21.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-02-05T04:39:21.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-02-05T04:39:21.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T04:39:21.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T04:39:21.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T04:39:21.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T04:39:22.182 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:39:22.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T04:39:24.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-02-05T04:39:24.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-02-05T04:39:24.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-02-05T04:39:24.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T04:39:24.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T04:39:24.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T04:39:24.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T04:39:24.462 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:39:24.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T04:39:26.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-02-05T04:39:26.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-02-05T04:39:26.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-02-05T04:39:26.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T04:39:26.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T04:39:26.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T04:39:26.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T04:39:26.738 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:39:26.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-02-05T04:39:28.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-02-05T04:39:28.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-02-05T04:39:28.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-02-05T04:39:28.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-02-05T04:39:28.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-02-05T04:39:28.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-02-05T04:39:28.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-02-05T04:39:29.008 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:39:29.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-02-05T04:39:29.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-02-05T04:39:29.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-02-05T04:39:29.276 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:39:29.276 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-02-05T04:39:24.335639+0000 24'1300 27:1322 [1,0,2,3] 1 [1,0] 1 0'0 2024-02-05T04:36:44.497716+0000 0'0 2024-02-05T04:36:44.497716+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:54:12.176343+0000 0 0 2024-02-05T04:39:29.276 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:39:29.276 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:39:29.276 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:39:29.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-02-05T04:39:39.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-02-05T04:39:39.289 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:39:39.289 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:39:39.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:39:39.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:39:39.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:39:40.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:39:40.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-02-05T04:39:40.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-02-05T04:39:40.600 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-02-05T04:39:40.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-02-05T04:39:40.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-02-05T04:39:40.605 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:30.082+0000 7f0e65825640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.605 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:32.142+0000 7f0e65825640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.605 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:38.142+0000 7f0e65825640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.605 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:39.136+0000 7f0e65825640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-02-05T04:39:40.618 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:26.390+0000 7fa7818c6640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.618 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:29.446+0000 7fa7818c6640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.618 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:32.411+0000 7fa7818c6640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.618 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:37.491+0000 7fa7818c6640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-02-05T04:39:40.621 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:27.122+0000 7fed2cd41640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.621 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:28.110+0000 7fed2cd41640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.621 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:32.127+0000 7fed2cd41640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.621 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:34.138+0000 7fed2cd41640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.621 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:37.135+0000 7fed2cd41640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:25.286+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:26.261+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:27.213+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:30.224+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:32.231+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:33.275+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:36.362+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.624 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:39:39.404+0000 7f3d41ba2640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-02-05T04:39:40.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-02-05T04:39:40.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-02-05T04:39:40.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-02-05T04:39:40.625 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-02-05T04:39:40.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-02-05T04:39:40.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-02-05T04:39:40.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T04:39:40.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-02-05T04:39:40.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-02-05T04:39:40.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-02-05T04:39:40.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T04:39:40.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-02-05T04:39:40.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-02-05T04:39:40.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-02-05T04:39:40.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T04:39:40.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-02-05T04:39:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-02-05T04:39:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-02-05T04:39:40.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-02-05T04:39:40.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-02-05T04:39:40.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-02-05T04:39:40.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-02-05T04:39:40.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-02-05T04:39:40.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-02-05T04:39:40.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T04:39:40.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:39:40.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T04:39:40.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:39:40.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:39:40.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:39:40.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:39:40.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:39:40.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:39:40.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:39:40.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:39:40.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:39:40.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:39:40.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:39:40.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:39:40.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:39:40.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:39:40.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:39:40.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:39:40.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:39:40.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:39:40.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T04:39:40.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:39:40.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:40.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:40.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44176 2024-02-05T04:39:40.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:39:40.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:39:40.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-02-05T04:39:40.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-02-05T04:39:40.687 INFO:tasks.workunit.client.0.smithi188.stdout:TEST PASSED 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.44176 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-02-05T04:39:40.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:39:40.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:39:40.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:39:40.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:39:40.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:39:40.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:39:40.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:39:40.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:39:40.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:39:40.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:39:40.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:39:40.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:39:40.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:39:40.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T04:39:40.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:39:40.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:40.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:40.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44176 2024-02-05T04:39:40.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:39:40.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:39:40.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44176 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:39:40.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:39:40.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:39:40.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-02-05T04:39:40.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:39:41.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:39:41.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:39:41.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:39:41.051 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:39:41.051 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:41.051 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:41.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:39:41.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T04:39:41.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:41.245 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44176/ceph-mon.a.asok 2024-02-05T04:39:41.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:39:41.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44176/ceph-mon.a.asok config get fsid 2024-02-05T04:39:41.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44176/ceph-mon.a.asok 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:39:41.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44176/ceph-mon.a.asok config get mon_host 2024-02-05T04:39:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-02-05T04:39:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-02-05T04:39:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:39:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:39:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:39:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-02-05T04:39:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:39:42.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:39:42.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:39:42.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:39:42.211 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:39:42.211 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:42.212 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:42.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:39:42.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:39:42.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:39:42.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-02-05T04:39:42.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-02-05T04:39:42.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-02-05T04:39:42.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-02-05T04:39:42.239 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-02-05T04:39:42.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-02-05T04:39:42.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:42.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-02-05T04:39:42.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:39:42.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:39:42.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2663a989-1d31-4c58-b0bc-239b2bfe1e6b 2024-02-05T04:39:42.247 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 2663a989-1d31-4c58-b0bc-239b2bfe1e6b 2024-02-05T04:39:42.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2663a989-1d31-4c58-b0bc-239b2bfe1e6b' 2024-02-05T04:39:42.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:39:42.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOZsBl5GyTDxAAVzbAMQCITYkMbP8SM7yd9A== 2024-02-05T04:39:42.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOZsBl5GyTDxAAVzbAMQCITYkMbP8SM7yd9A=="}' 2024-02-05T04:39:42.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2663a989-1d31-4c58-b0bc-239b2bfe1e6b -i td/osd-recovery-scrub/0/new.json 2024-02-05T04:39:42.467 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:39:42.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-02-05T04:39:42.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCOZsBl5GyTDxAAVzbAMQCITYkMbP8SM7yd9A== --osd-uuid 2663a989-1d31-4c58-b0bc-239b2bfe1e6b 2024-02-05T04:39:42.513 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:42.513+0000 7f2ef780d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:42.518 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:42.518+0000 7f2ef780d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:42.519 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:42.519+0000 7f2ef780d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:42.520 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:42.519+0000 7f2ef780d4c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:39:42.520 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:42.519+0000 7f2ef780d4c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:39:42.520 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:42.519+0000 7f2ef780d4c0 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-02-05T04:39:42.611 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:42.610+0000 7f2ef780d4c0 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-02-05T04:39:48.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-02-05T04:39:48.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:39:48.008 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:39:48.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:39:48.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:39:48.389 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:39:48.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:39:48.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:39:48.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:39:48.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:39:48.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:39:48.420 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:48.420+0000 7efc59c0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:48.422 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:48.422+0000 7efc59c0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:48.423 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:48.423+0000 7efc59c0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:48.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:39:48.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:39:48.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:39:48.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:39:48.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:39:48.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:39:48.657 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:39:48.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:39:48.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:39:48.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:39:48.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:39:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:48.950+0000 7efc59c0e4c0 -1 Falling back to public interface 2024-02-05T04:39:49.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:39:49.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:39:49.924 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:39:49.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:39:49.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:39:49.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:39:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:39:51.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:39:51.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:39:51.202 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:39:51.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:39:51.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:39:51.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:39:51.286 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:51.286+0000 7efc59c0e4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:39:51.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:39:52.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:39:52.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:39:52.522 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:39:52.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:39:52.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:39:52.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:39:52.818 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2725744812,v1:127.0.0.1:6803/2725744812] [v2:127.0.0.1:6804/2725744812,v1:127.0.0.1:6805/2725744812] exists,up 2663a989-1d31-4c58-b0bc-239b2bfe1e6b 2024-02-05T04:39:52.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:39:52.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:39:52.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:39:52.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:39:52.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-02-05T04:39:52.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:39:52.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcbb2393-f384-4fc8-8a10-c273422f45f8 2024-02-05T04:39:52.826 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 dcbb2393-f384-4fc8-8a10-c273422f45f8 2024-02-05T04:39:52.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dcbb2393-f384-4fc8-8a10-c273422f45f8' 2024-02-05T04:39:52.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:39:52.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYZsBl74P9MRAAWybhVfatf8WgZcA9GgILKA== 2024-02-05T04:39:52.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYZsBl74P9MRAAWybhVfatf8WgZcA9GgILKA=="}' 2024-02-05T04:39:52.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcbb2393-f384-4fc8-8a10-c273422f45f8 -i td/osd-recovery-scrub/1/new.json 2024-02-05T04:39:53.148 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:39:53.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-02-05T04:39:53.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCYZsBl74P9MRAAWybhVfatf8WgZcA9GgILKA== --osd-uuid dcbb2393-f384-4fc8-8a10-c273422f45f8 2024-02-05T04:39:53.191 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:53.191+0000 7fd667e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:53.232 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:53.232+0000 7fd667e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:53.233 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:53.233+0000 7fd667e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:53.234 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:53.233+0000 7fd667e164c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:39:53.234 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:53.233+0000 7fd667e164c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:39:53.234 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:53.233+0000 7fd667e164c0 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-02-05T04:39:53.309 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:53.308+0000 7fd667e164c0 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-02-05T04:39:59.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-02-05T04:39:59.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:39:59.099 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:39:59.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:39:59.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:39:59.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:39:59.443 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:39:59.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:39:59.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:39:59.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:39:59.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:39:59.473 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:59.472+0000 7fcc0bc0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:59.474 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:59.474+0000 7fcc0bc0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:59.475 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:39:59.475+0000 7fcc0bc0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:39:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:39:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:39:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:39:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:39:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:39:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:39:59.711 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:39:59.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:39:59.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:39:59.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:39:59.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:00.013 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:00.013+0000 7fcc0bc0e4c0 -1 Falling back to public interface 2024-02-05T04:40:00.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:00.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:00.977 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:40:00.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:40:00.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:00.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:40:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:02.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:02.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:02.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:40:02.252 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:40:02.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:02.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:40:02.474 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:02.474+0000 7fcc0bc0e4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:40:02.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:03.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:03.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:03.558 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:40:03.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:40:03.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:03.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:40:03.874 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/602085251,v1:127.0.0.1:6811/602085251] [v2:127.0.0.1:6812/602085251,v1:127.0.0.1:6813/602085251] exists,up dcbb2393-f384-4fc8-8a10-c273422f45f8 2024-02-05T04:40:03.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-02-05T04:40:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:40:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:40:03.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:40:03.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:40:03.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-02-05T04:40:03.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:40:03.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1032cd17-4d3e-4bf3-bbfe-0ac6be432191 2024-02-05T04:40:03.882 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 1032cd17-4d3e-4bf3-bbfe-0ac6be432191 2024-02-05T04:40:03.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1032cd17-4d3e-4bf3-bbfe-0ac6be432191' 2024-02-05T04:40:03.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:40:03.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjZsBljC1eNRAA04ZasQAZlS0lHsbYdc6Q8g== 2024-02-05T04:40:03.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjZsBljC1eNRAA04ZasQAZlS0lHsbYdc6Q8g=="}' 2024-02-05T04:40:03.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1032cd17-4d3e-4bf3-bbfe-0ac6be432191 -i td/osd-recovery-scrub/2/new.json 2024-02-05T04:40:04.288 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:40:04.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-02-05T04:40:04.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCjZsBljC1eNRAA04ZasQAZlS0lHsbYdc6Q8g== --osd-uuid 1032cd17-4d3e-4bf3-bbfe-0ac6be432191 2024-02-05T04:40:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:04.332+0000 7f994f0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:04.373 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:04.372+0000 7f994f0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:04.374 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:04.373+0000 7f994f0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:04.374 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:04.374+0000 7f994f0164c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:40:04.374 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:04.374+0000 7f994f0164c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:40:04.374 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:04.374+0000 7f994f0164c0 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-02-05T04:40:04.448 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:04.447+0000 7f994f0164c0 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-02-05T04:40:10.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-02-05T04:40:10.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:40:10.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:40:10.401 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:40:10.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:40:10.746 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:40:10.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:40:10.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:10.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:40:10.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:40:10.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:40:10.776 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:10.776+0000 7f5ffdbe24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:10.778 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:10.778+0000 7f5ffdbe24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:10.779 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:10.779+0000 7f5ffdbe24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:11.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:40:11.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:40:11.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:40:11.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:40:11.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:40:11.015 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:40:11.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:11.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:40:11.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:11.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:40:11.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:11.302+0000 7f5ffdbe24c0 -1 Falling back to public interface 2024-02-05T04:40:12.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:12.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:12.282 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:40:12.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:40:12.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:12.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:40:12.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:13.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:13.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:13.596 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:40:13.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:40:13.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:13.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:40:13.724 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:13.724+0000 7f5ffdbe24c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:40:13.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:14.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:14.924 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:40:14.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:40:14.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:14.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:40:15.195 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2195225113,v1:127.0.0.1:6819/2195225113] [v2:127.0.0.1:6820/2195225113,v1:127.0.0.1:6821/2195225113] exists,up 1032cd17-4d3e-4bf3-bbfe-0ac6be432191 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:40:15.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:40:15.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:40:15.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:40:15.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:40:15.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:40:15.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:40:15.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:40:15.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-02-05T04:40:15.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:40:15.202 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 78633ef3-8148-47f0-973e-58811a84514c 2024-02-05T04:40:15.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=78633ef3-8148-47f0-973e-58811a84514c 2024-02-05T04:40:15.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 78633ef3-8148-47f0-973e-58811a84514c' 2024-02-05T04:40:15.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:40:15.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvZsBl8nXrDBAA1dykljIcJjZbGtXYB+fgUA== 2024-02-05T04:40:15.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvZsBl8nXrDBAA1dykljIcJjZbGtXYB+fgUA=="}' 2024-02-05T04:40:15.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 78633ef3-8148-47f0-973e-58811a84514c -i td/osd-recovery-scrub/3/new.json 2024-02-05T04:40:15.569 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:40:15.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-02-05T04:40:15.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCvZsBl8nXrDBAA1dykljIcJjZbGtXYB+fgUA== --osd-uuid 78633ef3-8148-47f0-973e-58811a84514c 2024-02-05T04:40:15.612 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:15.612+0000 7f71842124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:15.645 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:15.645+0000 7f71842124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:15.646 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:15.645+0000 7f71842124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:15.646 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:15.646+0000 7f71842124c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:40:15.646 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:15.646+0000 7f71842124c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:40:15.646 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:15.646+0000 7f71842124c0 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-02-05T04:40:15.720 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:15.719+0000 7f71842124c0 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-02-05T04:40:21.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-02-05T04:40:21.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:40:21.537 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T04:40:21.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T04:40:21.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:40:21.880 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T04:40:21.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T04:40:21.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:21.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:40:21.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:40:21.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:40:21.910 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:21.910+0000 7f78c100e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:21.912 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:21.912+0000 7f78c100e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:21.913 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:21.913+0000 7f78c100e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:40:22.150 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:40:22.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:22.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:40:22.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:22.433 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:22.433+0000 7f78c100e4c0 -1 Falling back to public interface 2024-02-05T04:40:23.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:23.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:23.426 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:40:23.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:40:23.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:23.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:40:23.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:24.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:24.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:24.702 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:40:24.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:40:24.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:24.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:40:24.846 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:24.845+0000 7f78c100e4c0 -1 osd.3 0 log_to_monitors true 2024-02-05T04:40:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:25.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:25.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:25.971 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:40:25.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:40:25.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:25.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T04:40:26.244 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2435549615,v1:127.0.0.1:6827/2435549615] [v2:127.0.0.1:6828/2435549615,v1:127.0.0.1:6829/2435549615] exists,up 78633ef3-8148-47f0-973e-58811a84514c 2024-02-05T04:40:26.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:40:26.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:40:26.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:40:26.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:40:26.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:26.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:40:26.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:40:26.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43743aa1-7dd7-41b9-8eb7-9def0fc7904b 2024-02-05T04:40:26.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 43743aa1-7dd7-41b9-8eb7-9def0fc7904b' 2024-02-05T04:40:26.253 INFO:tasks.workunit.client.0.smithi188.stdout:add osd4 43743aa1-7dd7-41b9-8eb7-9def0fc7904b 2024-02-05T04:40:26.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:40:26.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6ZsBlGdnYDxAAZ+9O2tQJcWxVT+r9hl8kvg== 2024-02-05T04:40:26.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6ZsBlGdnYDxAAZ+9O2tQJcWxVT+r9hl8kvg=="}' 2024-02-05T04:40:26.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43743aa1-7dd7-41b9-8eb7-9def0fc7904b -i td/osd-recovery-scrub/4/new.json 2024-02-05T04:40:26.587 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:40:26.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-02-05T04:40:26.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC6ZsBlGdnYDxAAZ+9O2tQJcWxVT+r9hl8kvg== --osd-uuid 43743aa1-7dd7-41b9-8eb7-9def0fc7904b 2024-02-05T04:40:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:26.628+0000 7f87dff2b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:26.669 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:26.669+0000 7f87dff2b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:26.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:26.670+0000 7f87dff2b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:26.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:26.670+0000 7f87dff2b4c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-05T04:40:26.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:26.670+0000 7f87dff2b4c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-05T04:40:26.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:26.670+0000 7f87dff2b4c0 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-02-05T04:40:26.745 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:26.745+0000 7f87dff2b4c0 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-02-05T04:40:32.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-02-05T04:40:32.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:40:32.227 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd4 key to auth repository 2024-02-05T04:40:32.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-02-05T04:40:32.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:40:32.594 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.4 2024-02-05T04:40:32.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-02-05T04:40:32.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:32.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:40:32.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:40:32.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:40:32.624 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:32.623+0000 7f716fe124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:32.626 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:32.626+0000 7f716fe124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:32.627 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:32.627+0000 7f716fe124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:40:32.865 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:40:32.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:32.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T04:40:33.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:33.148 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:33.148+0000 7f716fe124c0 -1 Falling back to public interface 2024-02-05T04:40:34.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:34.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:34.132 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:40:34.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:40:34.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:34.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T04:40:34.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:35.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:35.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:40:35.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:40:35.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:35.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T04:40:35.480 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:35.480+0000 7f716fe124c0 -1 osd.4 0 log_to_monitors true 2024-02-05T04:40:35.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:36.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:36.701 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:40:36.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:40:36.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:36.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T04:40:36.990 INFO:tasks.workunit.client.0.smithi188.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3010575450,v1:127.0.0.1:6835/3010575450] [v2:127.0.0.1:6836/3010575450,v1:127.0.0.1:6837/3010575450] exists,up 43743aa1-7dd7-41b9-8eb7-9def0fc7904b 2024-02-05T04:40:36.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:40:36.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:40:36.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:40:36.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:40:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:40:36.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:40:36.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-02-05T04:40:36.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:40:36.998 INFO:tasks.workunit.client.0.smithi188.stdout:add osd5 b9c99594-4a84-4e6e-8fd8-7ffa97469457 2024-02-05T04:40:36.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9c99594-4a84-4e6e-8fd8-7ffa97469457 2024-02-05T04:40:36.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 b9c99594-4a84-4e6e-8fd8-7ffa97469457' 2024-02-05T04:40:36.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:40:37.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFZsBl+vGjABAA4XgI/qU6GNGkv5E2wb2I1w== 2024-02-05T04:40:37.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFZsBl+vGjABAA4XgI/qU6GNGkv5E2wb2I1w=="}' 2024-02-05T04:40:37.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9c99594-4a84-4e6e-8fd8-7ffa97469457 -i td/osd-recovery-scrub/5/new.json 2024-02-05T04:40:37.355 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:40:37.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-02-05T04:40:37.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDFZsBl+vGjABAA4XgI/qU6GNGkv5E2wb2I1w== --osd-uuid b9c99594-4a84-4e6e-8fd8-7ffa97469457 2024-02-05T04:40:37.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:37.399+0000 7f88f64254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:37.448 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:37.447+0000 7f88f64254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:37.449 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:37.448+0000 7f88f64254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:37.449 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:37.449+0000 7f88f64254c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-05T04:40:37.449 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:37.449+0000 7f88f64254c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-05T04:40:37.449 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:37.449+0000 7f88f64254c0 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-02-05T04:40:37.515 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:37.514+0000 7f88f64254c0 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-02-05T04:40:43.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-02-05T04:40:43.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:40:43.755 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd5 key to auth repository 2024-02-05T04:40:43.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-02-05T04:40:43.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:40:44.099 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.5 2024-02-05T04:40:44.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-02-05T04:40:44.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:44.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:40:44.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:40:44.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:40:44.129 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:44.129+0000 7f51c759b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:44.131 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:44.131+0000 7f51c759b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:44.132 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:44.132+0000 7f51c759b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:44.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-02-05T04:40:44.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:40:44.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T04:40:44.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:40:44.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:40:44.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:44.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:40:44.380 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:40:44.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:44.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T04:40:44.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:44.663 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:44.663+0000 7f51c759b4c0 -1 Falling back to public interface 2024-02-05T04:40:45.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:45.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:45.644 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:40:45.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:40:45.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:45.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T04:40:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:46.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:46.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:46.920 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:40:46.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:40:46.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:46.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T04:40:47.035 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:47.035+0000 7f51c759b4c0 -1 osd.5 0 log_to_monitors true 2024-02-05T04:40:47.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:48.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:48.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:48.238 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:40:48.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:40:48.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:48.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T04:40:48.594 INFO:tasks.workunit.client.0.smithi188.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1482280826,v1:127.0.0.1:6843/1482280826] [v2:127.0.0.1:6844/1482280826,v1:127.0.0.1:6845/1482280826] exists,up b9c99594-4a84-4e6e-8fd8-7ffa97469457 2024-02-05T04:40:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:40:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:40:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:40:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:40:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:40:48.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:40:48.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0de48fc1-a6be-4deb-be05-44ce13a11b09 2024-02-05T04:40:48.601 INFO:tasks.workunit.client.0.smithi188.stdout:add osd6 0de48fc1-a6be-4deb-be05-44ce13a11b09 2024-02-05T04:40:48.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 0de48fc1-a6be-4deb-be05-44ce13a11b09' 2024-02-05T04:40:48.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:40:48.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQZsBloWi2JBAAmvN64/kviwZYfESAFa3nTA== 2024-02-05T04:40:48.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQZsBloWi2JBAAmvN64/kviwZYfESAFa3nTA=="}' 2024-02-05T04:40:48.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0de48fc1-a6be-4deb-be05-44ce13a11b09 -i td/osd-recovery-scrub/6/new.json 2024-02-05T04:40:48.983 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:40:48.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-02-05T04:40:48.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDQZsBloWi2JBAAmvN64/kviwZYfESAFa3nTA== --osd-uuid 0de48fc1-a6be-4deb-be05-44ce13a11b09 2024-02-05T04:40:49.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:49.026+0000 7f0de941f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:49.068 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:49.067+0000 7f0de941f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:49.069 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:49.068+0000 7f0de941f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:49.069 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:49.069+0000 7f0de941f4c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-05T04:40:49.069 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:49.069+0000 7f0de941f4c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-05T04:40:49.069 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:49.069+0000 7f0de941f4c0 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-02-05T04:40:49.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:49.134+0000 7f0de941f4c0 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-02-05T04:40:55.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-02-05T04:40:55.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:40:55.244 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd6 key to auth repository 2024-02-05T04:40:55.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-02-05T04:40:55.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:40:55.588 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.6 2024-02-05T04:40:55.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-02-05T04:40:55.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:40:55.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:40:55.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:40:55.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:40:55.619 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:55.618+0000 7fad263e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:55.621 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:55.620+0000 7fad263e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:55.622 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:55.621+0000 7fad263e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:40:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-02-05T04:40:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:40:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-02-05T04:40:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:40:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:40:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:40:55.857 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:40:55.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:55.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T04:40:56.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:56.149 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:56.149+0000 7fad263e24c0 -1 Falling back to public interface 2024-02-05T04:40:57.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:57.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:57.125 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:40:57.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:40:57.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:57.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T04:40:57.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:58.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:58.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:58.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:40:58.403 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:40:58.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:58.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T04:40:58.621 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:40:58.620+0000 7fad263e24c0 -1 osd.6 0 log_to_monitors true 2024-02-05T04:40:58.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:40:59.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:40:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:40:59.680 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:40:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:40:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:40:59.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T04:41:00.004 INFO:tasks.workunit.client.0.smithi188.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2848543873,v1:127.0.0.1:6851/2848543873] [v2:127.0.0.1:6852/2848543873,v1:127.0.0.1:6853/2848543873] exists,up 0de48fc1-a6be-4deb-be05-44ce13a11b09 2024-02-05T04:41:00.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:41:00.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:41:00.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:41:00.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:41:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-02-05T04:41:00.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b5976780-82b9-4c5e-b773-2bfe6c017cc1 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 b5976780-82b9-4c5e-b773-2bfe6c017cc1' 2024-02-05T04:41:00.011 INFO:tasks.workunit.client.0.smithi188.stdout:add osd7 b5976780-82b9-4c5e-b773-2bfe6c017cc1 2024-02-05T04:41:00.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:41:00.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcZsBlkS19ARAAqyQeYVtqQ1JddaeShfBAkw== 2024-02-05T04:41:00.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcZsBlkS19ARAAqyQeYVtqQ1JddaeShfBAkw=="}' 2024-02-05T04:41:00.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b5976780-82b9-4c5e-b773-2bfe6c017cc1 -i td/osd-recovery-scrub/7/new.json 2024-02-05T04:41:00.435 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:41:00.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-02-05T04:41:00.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDcZsBlkS19ARAAqyQeYVtqQ1JddaeShfBAkw== --osd-uuid b5976780-82b9-4c5e-b773-2bfe6c017cc1 2024-02-05T04:41:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:00.479+0000 7fb55d4ab4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:41:00.494 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:00.494+0000 7fb55d4ab4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:41:00.495 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:00.495+0000 7fb55d4ab4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:41:00.496 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:00.495+0000 7fb55d4ab4c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-05T04:41:00.496 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:00.495+0000 7fb55d4ab4c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-05T04:41:00.496 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:00.495+0000 7fb55d4ab4c0 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-02-05T04:41:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:00.586+0000 7fb55d4ab4c0 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-02-05T04:41:06.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-02-05T04:41:06.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:41:06.513 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd7 key to auth repository 2024-02-05T04:41:06.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-02-05T04:41:06.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:41:06.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-02-05T04:41:06.860 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.7 2024-02-05T04:41:06.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=81b9738d-de94-41d2-9848-1f69e2f351bd --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44176/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-02-05T04:41:06.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:41:06.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:41:06.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:41:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:06.889+0000 7f970ba0c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:41:06.891 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:06.891+0000 7f970ba0c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:41:06.892 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:06.892+0000 7f970ba0c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:41:07.129 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:41:07.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:41:07.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T04:41:07.417 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:07.417+0000 7f970ba0c4c0 -1 Falling back to public interface 2024-02-05T04:41:07.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:41:08.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:41:08.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:41:08.435 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:41:08.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:41:08.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:41:08.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T04:41:08.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:41:09.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:41:09.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:41:09.711 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:41:09.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:41:09.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:41:09.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T04:41:09.857 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:41:09.856+0000 7f970ba0c4c0 -1 osd.7 0 log_to_monitors true 2024-02-05T04:41:09.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:41:10.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:41:10.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:41:10.989 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:41:10.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:41:10.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:41:10.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T04:41:11.264 INFO:tasks.workunit.client.0.smithi188.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1831040249,v1:127.0.0.1:6859/1831040249] [v2:127.0.0.1:6860/1831040249,v1:127.0.0.1:6861/1831040249] exists,up b5976780-82b9-4c5e-b773-2bfe6c017cc1 2024-02-05T04:41:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:41:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:41:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:41:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-02-05T04:41:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-02-05T04:41:11.670 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T04:41:11.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:41:12.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-02-05T04:41:12.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:41:12.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:41:12.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:41:12.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:41:12.684 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:41:12.684 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:41:12.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:41:12.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:41:12.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:41:12.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:41:12.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:41:12.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:41:12.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:41:12.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:41:12.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:7' 2024-02-05T04:41:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:41:13.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:13.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:41:13.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-02-05T04:41:13.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-02-05T04:41:13.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-02-05T04:41:13.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:13.171 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:41:13.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-02-05T04:41:13.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-02-05T04:41:13.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975' 2024-02-05T04:41:13.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:13.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:41:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-05T04:41:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-05T04:41:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453' 2024-02-05T04:41:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:13.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T04:41:13.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-02-05T04:41:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-02-05T04:41:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931' 2024-02-05T04:41:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:13.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T04:41:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-02-05T04:41:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-02-05T04:41:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409' 2024-02-05T04:41:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:13.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T04:41:14.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-02-05T04:41:14.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-02-05T04:41:14.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887' 2024-02-05T04:41:14.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:14.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T04:41:14.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-02-05T04:41:14.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-02-05T04:41:14.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855364' 2024-02-05T04:41:14.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:41:14.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T04:41:14.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-05T04:41:14.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-05T04:41:14.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855364 7-171798691842' 2024-02-05T04:41:14.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:14.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-02-05T04:41:14.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:41:14.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-02-05T04:41:14.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:14.245 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836498 2024-02-05T04:41:14.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-02-05T04:41:14.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-02-05T04:41:14.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:41:14.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-02-05T04:41:14.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:14.526 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-02-05T04:41:14.526 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:14.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:41:14.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-02-05T04:41:14.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:14.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-02-05T04:41:14.529 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672975 2024-02-05T04:41:14.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-02-05T04:41:14.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:41:14.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-02-05T04:41:14.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:14.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-05T04:41:14.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:14.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:41:14.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-05T04:41:14.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:14.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-05T04:41:14.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-05T04:41:14.812 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509453 2024-02-05T04:41:14.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:41:15.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-02-05T04:41:15.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:15.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-02-05T04:41:15.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T04:41:15.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-02-05T04:41:15.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:15.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-02-05T04:41:15.092 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345931 2024-02-05T04:41:15.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-02-05T04:41:15.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T04:41:15.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-02-05T04:41:15.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:15.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-02-05T04:41:15.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:15.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T04:41:15.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-02-05T04:41:15.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:15.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-02-05T04:41:15.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-02-05T04:41:15.364 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.4 seq 107374182409 2024-02-05T04:41:15.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T04:41:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182409 -lt 107374182409 2024-02-05T04:41:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:15.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-02-05T04:41:15.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T04:41:15.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-02-05T04:41:15.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:15.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-02-05T04:41:15.634 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.5 seq 128849018887 2024-02-05T04:41:15.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-02-05T04:41:15.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T04:41:15.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-02-05T04:41:15.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:15.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855364 2024-02-05T04:41:15.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:15.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T04:41:15.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855364 2024-02-05T04:41:15.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:15.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-02-05T04:41:15.905 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.6 seq 150323855364 2024-02-05T04:41:15.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855364' 2024-02-05T04:41:15.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T04:41:16.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855363 -lt 150323855364 2024-02-05T04:41:16.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:41:17.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:41:17.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T04:41:17.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855364 2024-02-05T04:41:17.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:41:17.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-02-05T04:41:17.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:41:17.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T04:41:17.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-02-05T04:41:17.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:41:17.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-05T04:41:17.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-02-05T04:41:17.455 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.7 seq 171798691842 2024-02-05T04:41:17.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T04:41:17.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-02-05T04:41:17.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:41:17.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:41:17.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:41:18.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:41:18.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-02-05T04:41:18.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:41:18.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:41:18.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:41:18.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-02-05T04:41:18.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:41:18.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:41:18.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-02-05T04:41:18.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-02-05T04:41:18.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-02-05T04:41:18.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-02-05T04:41:18.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.44176 bs=1M count=50 2024-02-05T04:41:19.093 INFO:tasks.workunit.client.0.smithi188.stderr:50+0 records in 2024-02-05T04:41:19.094 INFO:tasks.workunit.client.0.smithi188.stderr:50+0 records out 2024-02-05T04:41:19.094 INFO:tasks.workunit.client.0.smithi188.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.140298 s, 374 MB/s 2024-02-05T04:41:19.094 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-02-05T04:41:19.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:19.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.44176 2024-02-05T04:41:20.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:20.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.44176 2024-02-05T04:41:22.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:22.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.44176 2024-02-05T04:41:23.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:23.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.44176 2024-02-05T04:41:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.44176 2024-02-05T04:41:26.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:26.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.44176 2024-02-05T04:41:28.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:28.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.44176 2024-02-05T04:41:29.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:29.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.44176 2024-02-05T04:41:31.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:31.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.44176 2024-02-05T04:41:32.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:32.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.44176 2024-02-05T04:41:34.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:34.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.44176 2024-02-05T04:41:35.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:35.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.44176 2024-02-05T04:41:37.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:37.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.44176 2024-02-05T04:41:38.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:38.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.44176 2024-02-05T04:41:40.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:40.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.44176 2024-02-05T04:41:41.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:41.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.44176 2024-02-05T04:41:43.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:43.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.44176 2024-02-05T04:41:44.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:44.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.44176 2024-02-05T04:41:46.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:46.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.44176 2024-02-05T04:41:47.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:47.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.44176 2024-02-05T04:41:49.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:49.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.44176 2024-02-05T04:41:51.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:51.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.44176 2024-02-05T04:41:52.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:52.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.44176 2024-02-05T04:41:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.44176 2024-02-05T04:41:55.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:55.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.44176 2024-02-05T04:41:57.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:57.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.44176 2024-02-05T04:41:58.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:41:58.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.44176 2024-02-05T04:42:00.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:00.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.44176 2024-02-05T04:42:01.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:01.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.44176 2024-02-05T04:42:03.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:03.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.44176 2024-02-05T04:42:04.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:04.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.44176 2024-02-05T04:42:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.44176 2024-02-05T04:42:07.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:07.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.44176 2024-02-05T04:42:09.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:09.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.44176 2024-02-05T04:42:10.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:10.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.44176 2024-02-05T04:42:11.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:11.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.44176 2024-02-05T04:42:13.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:13.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.44176 2024-02-05T04:42:15.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:15.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.44176 2024-02-05T04:42:16.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:16.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.44176 2024-02-05T04:42:18.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-02-05T04:42:18.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.44176 2024-02-05T04:42:19.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.44176 2024-02-05T04:42:19.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-02-05T04:42:19.984 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 3 2024-02-05T04:42:19.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-05T04:41:12.772003+0000 47'26 47:51 [5] 5 [5] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284650+0000 47'13 47:37 [3] 3 [3] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.1d 2 0 0 0 0 81788928 0 0 20 0 20 active+clean 2024-02-05T04:41:13.384464+0000 47'20 47:32 [1] 1 [1] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284062+0000 47'13 47:28 [1] 1 [1] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284063+0000 47'13 47:28 [1] 1 [1] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:41:13.284316+0000 0'0 47:15 [1] 1 [1] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-05T04:41:12.660669+0000 47'26 47:42 [5] 5 [5] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284099+0000 47'13 47:30 [6] 6 [6] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.250 INFO:tasks.workunit.client.0.smithi188.stdout:1.6 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284372+0000 47'13 47:30 [6] 6 [6] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-02-05T04:41:13.233497+0000 47'39 47:57 [4] 4 [4] 4 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-02-05T04:41:13.284051+0000 47'52 47:70 [3] 3 [3] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-02-05T04:41:13.360910+0000 47'52 47:70 [4] 4 [4] 4 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.402365+0000 47'13 47:34 [0] 0 [0] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:41:13.284054+0000 0'0 47:16 [3] 3 [3] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.660660+0000 47'13 47:38 [5] 5 [5] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.402198+0000 47'13 47:26 [0] 0 [0] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.838834+0000 47'13 47:35 [7] 7 [7] 7 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.735289+0000 47'13 47:26 [7] 7 [7] 7 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284418+0000 47'13 47:27 [1] 1 [1] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284675+0000 47'13 47:27 [6] 6 [6] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.251 INFO:tasks.workunit.client.0.smithi188.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.771926+0000 47'13 47:28 [5] 5 [5] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.402444+0000 47'13 47:28 [0] 0 [0] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.12 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-02-05T04:41:12.250735+0000 47'26 47:40 [0] 0 [0] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.361020+0000 47'13 47:27 [4] 4 [4] 4 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:41:12.710564+0000 0'0 47:13 [2] 2 [2] 2 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-02-05T04:41:13.284493+0000 47'39 47:52 [1] 1 [1] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:41:13.384478+0000 0'0 47:13 [6] 6 [6] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.402103+0000 47'13 47:27 [0] 0 [0] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:41:12.771845+0000 0'0 47:14 [5] 5 [5] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:13.284530+0000 47'13 47:27 [3] 3 [3] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:41:13.284573+0000 0'0 47:13 [1] 1 [1] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-02-05T04:41:12.402511+0000 47'13 47:27 [0] 0 [0] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:20.252 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:42:20.253 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:42:20.253 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:42:20.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-02-05T04:42:20.379 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:42:20.379 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_recovery_max_active": "0" 2024-02-05T04:42:20.380 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:42:20.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-02-05T04:42:20.647 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:42:20.647 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_recovery_max_active_hdd": "3" 2024-02-05T04:42:20.647 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:42:20.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-02-05T04:42:20.900 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:42:20.901 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_recovery_max_active_ssd": "10" 2024-02-05T04:42:20.901 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:42:20.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-02-05T04:42:20.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-05T04:42:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-05T04:42:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-05T04:42:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-05T04:42:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-05T04:42:21.316 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:42:21.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-02-05T04:42:21.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-05T04:42:23.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-02-05T04:42:23.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-02-05T04:42:23.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-02-05T04:42:23.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-05T04:42:23.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-05T04:42:23.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-05T04:42:23.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-05T04:42:23.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-05T04:42:23.596 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:42:23.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-02-05T04:42:23.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-05T04:42:25.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-02-05T04:42:25.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-02-05T04:42:25.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-02-05T04:42:25.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-05T04:42:25.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-05T04:42:25.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-05T04:42:25.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-05T04:42:25.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-05T04:42:25.874 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:42:25.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-02-05T04:42:25.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-02-05T04:42:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-02-05T04:42:27.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-02-05T04:42:27.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-02-05T04:42:27.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-02-05T04:42:27.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-02-05T04:42:27.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-02-05T04:42:27.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-02-05T04:42:27.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-02-05T04:42:28.155 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:42:28.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-02-05T04:42:28.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-02-05T04:42:28.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-02-05T04:42:22.154276+0000 47'26 50:61 [5,0,2] 5 [5,0,2] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.920248+0000 47'13 50:47 [3,0,7] 3 [3,0,7] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-02-05T04:42:21.953727+0000 47'26 50:48 [1,3,0] 1 [1,3,0] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.920321+0000 47'13 50:37 [1,0,2] 1 [1,0,2] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.953411+0000 47'13 50:37 [1,6,2] 1 [1,6,2] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:42:21.862043+0000 0'0 50:26 [1,4,5] 1 [1,4,5] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-02-05T04:42:22.276833+0000 47'26 50:51 [5,4,3] 5 [5,4,3] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-02-05T04:42:21.954713+0000 47'13 50:41 [6,7,3] 6 [6,7,3] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.419 INFO:tasks.workunit.client.0.smithi188.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.953886+0000 47'13 50:40 [6,7,3] 6 [6,7,3] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-02-05T04:42:21.777050+0000 47'39 50:66 [4,2,6] 4 [4,2,6] 4 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-02-05T04:42:22.154328+0000 47'52 50:79 [3,0,6] 3 [3,0,6] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-02-05T04:42:22.277131+0000 47'52 50:79 [4,1,2] 4 [4,1,2] 4 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:22.087672+0000 47'13 50:44 [0,7,1] 0 [0,7,1] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:42:21.776369+0000 0'0 50:26 [3,0,4] 3 [3,0,4] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:22.355888+0000 47'13 50:48 [5,0,4] 5 [5,0,4] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.861186+0000 47'13 50:36 [0,3,4] 0 [0,3,4] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.591680+0000 47'13 50:46 [7,0,2] 7 [7,0,2] 7 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.861082+0000 47'13 50:35 [7,4,2] 7 [7,4,2] 7 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.861739+0000 47'13 50:36 [1,4,2] 1 [1,4,2] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.420 INFO:tasks.workunit.client.0.smithi188.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:22.355963+0000 47'13 50:36 [6,0,3] 6 [6,0,3] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:22.087901+0000 47'13 50:37 [5,7,6] 5 [5,7,6] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.860780+0000 47'13 50:37 [0,1,3] 0 [0,1,3] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-02-05T04:42:21.684656+0000 47'26 50:50 [0,1,6] 0 [0,1,6] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:21.860920+0000 47'13 50:36 [4,6,3] 4 [4,6,3] 4 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:42:21.919603+0000 0'0 50:23 [2,0,4] 2 [2,0,4] 2 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-02-05T04:42:21.920215+0000 47'39 50:61 [1,5,3] 1 [1,5,3] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:42:21.742846+0000 0'0 50:24 [6,0,7] 6 [6,0,7] 6 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:22.087816+0000 47'13 50:36 [0,4,5] 0 [0,4,5] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:42:21.953475+0000 0'0 50:25 [5,2,3] 5 [5,2,3] 5 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-02-05T04:42:22.154319+0000 47'13 50:35 [3,0,4] 3 [3,0,4] 3 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-02-05T04:42:21.685110+0000 0'0 50:24 [1,7,4] 1 [1,7,4] 1 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-02-05T04:42:21.685799+0000 47'13 50:37 [0,1,2] 0 [0,1,2] 0 0'0 2024-02-05T04:41:11.579353+0000 0'0 2024-02-05T04:41:11.579353+0000 0 0 periodic scrub scheduled @ 2024-02-06T04:41:11.579353+0000 0 0 2024-02-05T04:42:28.421 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:42:28.422 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:42:28.422 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:42:28.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-02-05T04:42:28.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-02-05T04:42:28.430 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-02-05T04:42:28.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-02-05T04:42:28.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65651"' 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65651' 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65654: /' 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65656"' 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65656' 2024-02-05T04:42:28.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65661: /' 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65664"' 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65664' 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-02-05T04:42:28.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65668: /' 2024-02-05T04:42:28.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-02-05T04:42:28.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65671"' 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65671' 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65676"' 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65676' 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65675: /' 2024-02-05T04:42:28.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65680: /' 2024-02-05T04:42:28.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-02-05T04:42:28.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65686"' 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65686' 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-02-05T04:42:28.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65688: /' 2024-02-05T04:42:28.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-02-05T04:42:28.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-02-05T04:42:28.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65693"' 2024-02-05T04:42:28.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65693' 2024-02-05T04:42:28.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65696: /' 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65697"' 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65697' 2024-02-05T04:42:28.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-02-05T04:42:28.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-02-05T04:42:28.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-02-05T04:42:28.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65703"' 2024-02-05T04:42:28.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65703' 2024-02-05T04:42:28.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-02-05T04:42:28.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-02-05T04:42:28.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-02-05T04:42:28.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65711"' 2024-02-05T04:42:28.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65711' 2024-02-05T04:42:28.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65708: /' 2024-02-05T04:42:28.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-02-05T04:42:28.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65700: /' 2024-02-05T04:42:28.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-02-05T04:42:28.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-02-05T04:42:28.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65718"' 2024-02-05T04:42:28.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65718' 2024-02-05T04:42:28.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65714: /' 2024-02-05T04:42:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-02-05T04:42:28.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-02-05T04:42:28.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-02-05T04:42:28.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65721"' 2024-02-05T04:42:28.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65721' 2024-02-05T04:42:28.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-02-05T04:42:28.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-02-05T04:42:28.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65729"' 2024-02-05T04:42:28.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65729' 2024-02-05T04:42:28.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65724: /' 2024-02-05T04:42:28.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-02-05T04:42:28.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-02-05T04:42:28.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-02-05T04:42:28.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65728: /' 2024-02-05T04:42:28.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-02-05T04:42:28.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65737"' 2024-02-05T04:42:28.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65737' 2024-02-05T04:42:28.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-02-05T04:42:28.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-02-05T04:42:28.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-02-05T04:42:28.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65742"' 2024-02-05T04:42:28.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65742' 2024-02-05T04:42:28.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65734: /' 2024-02-05T04:42:28.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-02-05T04:42:28.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-02-05T04:42:28.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65750"' 2024-02-05T04:42:28.526 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65750' 2024-02-05T04:42:28.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-02-05T04:42:28.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65747: /' 2024-02-05T04:42:28.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65749: /' 2024-02-05T04:42:28.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-02-05T04:42:28.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-02-05T04:42:28.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-02-05T04:42:28.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65754"' 2024-02-05T04:42:28.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65754' 2024-02-05T04:42:28.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65757: /' 2024-02-05T04:42:28.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-02-05T04:42:28.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65761: /' 2024-02-05T04:42:28.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-02-05T04:42:28.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-02-05T04:42:28.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65765"' 2024-02-05T04:42:28.556 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65765' 2024-02-05T04:42:28.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-02-05T04:42:28.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-02-05T04:42:28.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-02-05T04:42:28.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65771"' 2024-02-05T04:42:28.572 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65771' 2024-02-05T04:42:28.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-02-05T04:42:28.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-02-05T04:42:28.579 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-02-05T04:42:28.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-02-05T04:42:28.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65782"' 2024-02-05T04:42:28.580 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65782' 2024-02-05T04:42:28.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65776: /' 2024-02-05T04:42:28.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65779: /' 2024-02-05T04:42:28.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-02-05T04:42:28.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-02-05T04:42:28.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65789"' 2024-02-05T04:42:28.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65789' 2024-02-05T04:42:28.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-02-05T04:42:28.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-02-05T04:42:28.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65792"' 2024-02-05T04:42:28.603 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65792' 2024-02-05T04:42:28.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-02-05T04:42:28.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-02-05T04:42:28.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65794"' 2024-02-05T04:42:28.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65794' 2024-02-05T04:42:28.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-02-05T04:42:28.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-02-05T04:42:28.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-02-05T04:42:28.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-02-05T04:42:28.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65801"' 2024-02-05T04:42:28.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65801' 2024-02-05T04:42:28.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-02-05T04:42:28.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65800: /' 2024-02-05T04:42:28.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65806: /' 2024-02-05T04:42:28.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-02-05T04:42:28.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65799: /' 2024-02-05T04:42:28.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-02-05T04:42:28.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-02-05T04:42:28.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65823"' 2024-02-05T04:42:28.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65823' 2024-02-05T04:42:28.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65810: /' 2024-02-05T04:42:28.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-02-05T04:42:28.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65815: /' 2024-02-05T04:42:28.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-02-05T04:42:28.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65830"' 2024-02-05T04:42:28.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65830' 2024-02-05T04:42:28.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-02-05T04:42:28.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65832: /' 2024-02-05T04:42:28.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-02-05T04:42:28.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-02-05T04:42:28.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-02-05T04:42:28.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65846"' 2024-02-05T04:42:28.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65846' 2024-02-05T04:42:28.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65836: /' 2024-02-05T04:42:28.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-02-05T04:42:28.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-02-05T04:42:28.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-02-05T04:42:28.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65864"' 2024-02-05T04:42:28.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65864' 2024-02-05T04:42:28.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-02-05T04:42:28.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-02-05T04:42:28.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-02-05T04:42:28.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65890"' 2024-02-05T04:42:28.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65890' 2024-02-05T04:42:28.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65868: /' 2024-02-05T04:42:28.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-02-05T04:42:28.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-02-05T04:42:28.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65903"' 2024-02-05T04:42:28.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65903' 2024-02-05T04:42:28.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-02-05T04:42:28.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65893: /' 2024-02-05T04:42:28.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65896: /' 2024-02-05T04:42:28.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-02-05T04:42:28.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-02-05T04:42:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-02-05T04:42:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65920"' 2024-02-05T04:42:28.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65920' 2024-02-05T04:42:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-02-05T04:42:28.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-02-05T04:42:28.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65919: /' 2024-02-05T04:42:28.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-02-05T04:42:28.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 65942"' 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 65942' 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 65651 65656 65664 65671 65676 65686 65693 65697 65703 65711 65718 65721 65729 65737 65742 65750 65754 65765 65771 65782 65789 65792 65794 65801 65823 65830 65846 65864 65890 65903 65920 65942' 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:28.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65651 2024-02-05T04:42:28.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-02-05T04:42:28.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65941: /' 2024-02-05T04:42:28.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-02-05T04:42:28.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/65972: /' 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:34.706 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: instructing pg 1.7 on osd.6 to scrub 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: dumped pgs 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-02-05T04:42:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-02-05T04:42:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | se65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: instructing pg 1.2 on osd.0 to scrub 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: dumped pgs 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:65668: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:35.370 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-02-05T04:42:35.370 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-02-05T04:42:35.370 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: instructing pg 1.8 on osd.5 to scrub 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:35.371 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-02-05T04:42:35.372 INFO:tasks.workunit.client.0.smithi188.stderr:65708: dumped pgs 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-02-05T04:42:35.373 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:35.374 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:35.374 INFO:tasks.workunit.client.0.smithi188.stderr:65708: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-02-05T04:42:35.374 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:35.374 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:35.374 INFO:tasks.workunit.client.0.smithi188.stderr:65708: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:35.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.011 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-02-05T04:42:36.011 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-02-05T04:42:36.011 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-02-05T04:42:36.011 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: instructing pg 1.1e on osd.3 to scrub 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-02-05T04:42:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: dumped pgs 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-02-05T04:42:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:65941: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.014 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.014 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.014 INFO:tasks.workunit.client.0.smithi188.stderr:65941: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.144 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-02-05T04:42:36.144 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-02-05T04:42:36.144 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: instructing pg 1.9 on osd.1 to scrub 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-02-05T04:42:36.145 INFO:tasks.workunit.client.0.smithi188.stderr:65714: dumped pgs 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:42:32.854208+0000 '>' '' 2024-02-05T04:42:36.146 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:65714: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.269 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-02-05T04:42:36.269 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-02-05T04:42:36.269 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-02-05T04:42:36.269 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.269 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: instructing pg 1.6 on osd.6 to scrub 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: dumped pgs 2024-02-05T04:42:36.270 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.271 INFO:tasks.workunit.client.0.smithi188.stderr:65696: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.295 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-02-05T04:42:36.296 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-02-05T04:42:36.296 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-02-05T04:42:36.296 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.296 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-02-05T04:42:36.297 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.297 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.297 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.297 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-02-05T04:42:36.298 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.298 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.298 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.298 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-02-05T04:42:36.298 INFO:tasks.workunit.client.0.smithi188.stderr:65724: instructing pg 1.a on osd.1 to scrub 2024-02-05T04:42:36.298 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: dumped pgs 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.299 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:65724: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.502 INFO:tasks.workunit.client.0.smithi188.stderr:lect(.pgid=="1.7") | .last_scrub_stamp' 2024-02-05T04:42:36.502 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.502 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.502 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-02-05T04:42:36.502 INFO:tasks.workunit.client.0.smithi188.stderr:65700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-02-05T04:42:36.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-02-05T04:42:36.678 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-02-05T04:42:36.678 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-02-05T04:42:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:65779: instructing pg 1.12 on osd.0 to scrub 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: dumped pgs 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:65779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:65779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.730 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: instructing pg 1.1b on osd.0 to scrub 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: dumped pgs 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.732 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65893: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: g65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.743 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: instructing pg 1.1a on osd.1 to scrub 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: dumped pgs 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.744 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:42:33.543764+0000 '>' '' 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:65868: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.892 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-02-05T04:42:36.892 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-02-05T04:42:36.892 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T04:42:36.892 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: instructing pg 1.1 on osd.5 to scrub 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: dumped pgs 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.893 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:65661: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.900 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: instructing pg 1.0 on osd.7 to scrub 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: dumped pgs 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.901 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:65654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65656 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:36.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:36.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:36.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65664 2024-02-05T04:42:36.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:36.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:36.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:36.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:36.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65671 2024-02-05T04:42:36.955 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-02-05T04:42:36.955 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-02-05T04:42:36.955 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T04:42:36.955 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: instructing pg 1.4 on osd.3 to scrub 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: dumped pgs 2024-02-05T04:42:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:65680: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: instructing pg 1.e on osd.1 to scrub 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: dumped pgs 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:36.984 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:65749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:65749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:36.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.029 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-02-05T04:42:37.029 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-02-05T04:42:37.029 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-02-05T04:42:37.029 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: instructing pg 1.d on osd.7 to scrub 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: dumped pgs 2024-02-05T04:42:37.030 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:65747: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.066 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: instructing pg 1.15 on osd.1 to scrub 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: dumped pgs 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-02-05T04:42:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-02-05T04:42:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:65800: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-02-05T04:42:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:65800: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.159 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-02-05T04:42:37.159 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-02-05T04:42:37.159 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: instructing pg 1.3 on osd.4 to scrub 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: dumped pgs 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.162 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.162 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.162 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T04:42:37.162 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:42:37.162 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.162 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:42:37.162 INFO:tasks.workunit.client.0.smithi188.stderr:65675: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:65675: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65676 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65686 2024-02-05T04:42:37.313 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: instructing pg 1.1d on osd.1 to scrub 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: dumped pgs 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-02-05T04:42:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-02-05T04:42:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-02-05T04:42:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:65919: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:65919: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.318 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-02-05T04:42:37.318 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-02-05T04:42:37.318 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-02-05T04:42:37.318 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.318 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.319 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-02-05T04:42:37.319 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.319 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-02-05T04:42:37.319 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.319 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.320 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.320 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.320 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-02-05T04:42:37.320 INFO:tasks.workunit.client.0.smithi188.stderr:65799: instructing pg 1.13 on osd.4 to scrub 2024-02-05T04:42:37.320 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-02-05T04:42:37.320 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.321 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.321 INFO:tasks.workunit.client.0.smithi188.stderr:65799: dumped pgs 2024-02-05T04:42:37.321 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.321 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-02-05T04:42:37.321 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.321 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.322 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.322 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.322 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.322 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.322 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-02-05T04:42:37.322 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-02-05T04:42:37.323 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.323 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-02-05T04:42:37.323 INFO:tasks.workunit.client.0.smithi188.stderr:65799: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.323 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.323 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.323 INFO:tasks.workunit.client.0.smithi188.stderr:65799: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.343 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-02-05T04:42:37.344 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-02-05T04:42:37.344 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-02-05T04:42:37.344 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.344 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.344 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-02-05T04:42:37.344 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.345 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.345 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-02-05T04:42:37.345 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.345 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.345 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.345 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-02-05T04:42:37.346 INFO:tasks.workunit.client.0.smithi188.stderr:65810: instructing pg 1.14 on osd.2 to scrub 2024-02-05T04:42:37.346 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-02-05T04:42:37.346 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.346 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.346 INFO:tasks.workunit.client.0.smithi188.stderr:65810: dumped pgs 2024-02-05T04:42:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-02-05T04:42:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.348 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.348 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.348 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-02-05T04:42:37.348 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-02-05T04:42:37.348 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-02-05T04:42:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:65810: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:42:33.539902+0000 '>' '' 2024-02-05T04:42:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:65810: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.386 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-02-05T04:42:37.386 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-02-05T04:42:37.386 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-02-05T04:42:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-02-05T04:42:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-02-05T04:42:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-02-05T04:42:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:65896: instructing pg 1.1c on osd.0 to scrub 2024-02-05T04:42:37.389 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.389 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.389 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-02-05T04:42:37.389 INFO:tasks.workunit.client.0.smithi188.stderr:65896: dumped pgs 2024-02-05T04:42:37.389 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.389 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-02-05T04:42:37.390 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.390 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.390 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.390 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.390 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.390 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.391 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-02-05T04:42:37.391 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-02-05T04:42:37.391 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.391 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.391 INFO:tasks.workunit.client.0.smithi188.stderr:65896: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-02-05T04:42:37.391 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.392 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.392 INFO:tasks.workunit.client.0.smithi188.stderr:65896: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.534 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-02-05T04:42:37.534 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-02-05T04:42:37.534 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-02-05T04:42:37.534 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.534 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: instructing pg 1.16 on osd.6 to scrub 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: dumped pgs 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-02-05T04:42:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.537 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.537 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.537 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.537 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.537 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.537 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-02-05T04:42:37.537 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-02-05T04:42:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-02-05T04:42:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:65806: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:65806: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.550 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-02-05T04:42:37.550 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-02-05T04:42:37.550 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-02-05T04:42:37.550 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.550 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-02-05T04:42:37.550 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: instructing pg 1.19 on osd.3 to scrub 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: dumped pgs 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.551 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.552 INFO:tasks.workunit.client.0.smithi188.stderr:65836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.628 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-02-05T04:42:37.628 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-02-05T04:42:37.628 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-02-05T04:42:37.628 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.628 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-02-05T04:42:37.628 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: instructing pg 1.b on osd.1 to scrub 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: dumped pgs 2024-02-05T04:42:37.629 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.630 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.631 INFO:tasks.workunit.client.0.smithi188.stderr:65728: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.680 INFO:tasks.workunit.client.0.smithi188.stderr:et_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.681 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.681 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.681 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-02-05T04:42:37.681 INFO:tasks.workunit.client.0.smithi188.stderr:65893: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-02-05T04:42:37.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-02-05T04:42:37.687 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: instructing pg 1.18 on osd.5 to scrub 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: dumped pgs 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.688 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:65832: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.699 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: instructing pg 1.f on osd.6 to scrub 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: dumped pgs 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.700 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:65757: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.702 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: instructing pg 1.11 on osd.0 to scrub 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: dumped pgs 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-02-05T04:42:37.703 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-02-05T04:42:37.704 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.704 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-02-05T04:42:37.704 INFO:tasks.workunit.client.0.smithi188.stderr:65776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.704 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.704 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.704 INFO:tasks.workunit.client.0.smithi188.stderr:65776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: instructing pg 1.17 on osd.0 to scrub 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: dumped pgs 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:65815: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-02-05T04:42:37.775 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: instructing pg 1.1f on osd.5 to scrub 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: dumped pgs 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.776 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:65972: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.816 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-02-05T04:42:37.816 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-02-05T04:42:37.816 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-02-05T04:42:37.816 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: instructing pg 1.5 on osd.4 to scrub 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-02-05T04:42:37.817 INFO:tasks.workunit.client.0.smithi188.stderr:65688: dumped pgs 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-02-05T04:42:37.818 INFO:tasks.workunit.client.0.smithi188.stderr:65688: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:65688: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65693 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65697 2024-02-05T04:42:37.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-02-05T04:42:37.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-02-05T04:42:37.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65703 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65711 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65718 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65721 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65729 2024-02-05T04:42:37.868 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-02-05T04:42:37.868 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-02-05T04:42:37.868 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-02-05T04:42:37.868 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.868 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: instructing pg 1.c on osd.3 to scrub 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: dumped pgs 2024-02-05T04:42:37.869 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.870 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:65734: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65737 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65742 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65750 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65754 2024-02-05T04:42:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-02-05T04:42:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-02-05T04:42:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-02-05T04:42:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: instructing pg 1.10 on osd.5 to scrub 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-02-05T04:42:37.882 INFO:tasks.workunit.client.0.smithi188.stderr:65761: dumped pgs 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-02-05T04:41:11.579353+0000 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-02-05T04:41:11.579353+0000 '>' '' 2024-02-05T04:42:37.883 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:65761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65765 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65771 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65782 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65789 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65792 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65794 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65801 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65823 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65830 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65846 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65864 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65890 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65903 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65920 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 65942 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-02-05T04:42:37.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/mon.a.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.5.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.6.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.1.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.0.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.4.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.3.pid 2024-02-05T04:42:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.7.pid 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-recovery-scrub/osd.2.pid' 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.2.pid 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='52772 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:60684 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:62060 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:55178 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:52856 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:53802 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:59308 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:57932 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:63436 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:56556' 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 52772 60684 62060 55178 52856 53802 59308 57932 63436 56556 2024-02-05T04:42:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-02-05T04:42:37.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:42:37.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:42:37.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:42:37.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:42:37.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:42:43.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:42:43.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-02-05T04:42:43.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-02-05T04:42:43.195 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-02-05T04:42:43.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-02-05T04:42:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-02-05T04:42:43.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-02-05T04:42:43.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-02-05T04:42:43.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-02-05T04:42:43.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-02-05T04:42:43.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-02-05T04:42:43.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-02-05T04:42:43.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-02-05T04:42:43.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-02-05T04:42:43.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-02-05T04:42:43.227 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-02-05T04:42:43.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-02-05T04:42:43.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-02-05T04:42:43.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-02-05T04:42:43.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-02-05T04:42:43.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-02-05T04:42:43.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-02-05T04:42:43.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-02-05T04:42:43.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-02-05T04:42:43.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T04:42:43.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:42:43.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:42:43.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:42:43.262 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:42:43.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:42:43.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:42:43.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:42:43.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:42:43.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:42:43.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:42:43.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:42:43.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:42:43.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:42:43.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T04:42:43.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:42:43.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:43.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:42:43.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44176 2024-02-05T04:42:43.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:42:43.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:42:43.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-02-05T04:42:43.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-02-05T04:42:43.293 INFO:tasks.workunit.client.0.smithi188.stdout:TEST PASSED 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:42:43.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:42:43.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:42:43.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:42:43.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:42:43.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:42:43.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:42:43.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:42:43.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:42:43.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:42:43.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:42:43.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:42:43.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T04:42:43.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-02-05T04:42:43.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:42:43.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:43.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44176 2024-02-05T04:42:43.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44176 2024-02-05T04:42:43.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:42:43.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:42:43.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T04:42:43.304 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T04:42:43.304 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T04:42:43.332 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-02-05T04:42:43.333 DEBUG:teuthology.orchestra.run.smithi188:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-02-05T04:42:43.393 INFO:tasks.workunit.client.0.smithi188.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T04:42:43.397 INFO:tasks.workunit.client.0.smithi188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T04:42:43.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-02-05T04:42:43.398 INFO:tasks.workunit.client.0.smithi188.stdout:This test is disabled 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:42:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:42:43.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:42:43.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:42:43.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:42:43.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:42:43.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:42:43.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:42:43.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:42:43.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:42:43.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:42:43.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:42:43.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T04:42:43.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-02-05T04:42:43.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:42:43.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:43.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69142 2024-02-05T04:42:43.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69142 2024-02-05T04:42:43.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:42:43.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:42:43.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T04:42:43.407 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T04:42:43.407 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T04:42:43.473 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-02-05T04:42:43.473 DEBUG:teuthology.orchestra.run.smithi188:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-02-05T04:42:43.534 INFO:tasks.workunit.client.0.smithi188.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-02-05T04:42:43.534 INFO:tasks.workunit.client.0.smithi188.stderr:++ TIMEOUT=300 2024-02-05T04:42:43.534 INFO:tasks.workunit.client.0.smithi188.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-02-05T04:42:43.534 INFO:tasks.workunit.client.0.smithi188.stderr:++ MAX_TIMEOUT=15 2024-02-05T04:42:43.534 INFO:tasks.workunit.client.0.smithi188.stderr:++ PG_NUM=4 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ TMPDIR=/tmp 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ type xmlstarlet 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ XMLSTARLET=xmlstarlet 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:+++ uname 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ '[' Linux = FreeBSD ']' 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ SED=sed 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:++ AWK=awk 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:+++ stty -a 2024-02-05T04:42:43.535 INFO:tasks.workunit.client.0.smithi188.stderr:+++ head -1 2024-02-05T04:42:43.536 INFO:tasks.workunit.client.0.smithi188.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-02-05T04:42:43.536 INFO:tasks.workunit.client.0.smithi188.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T04:42:43.537 INFO:tasks.workunit.client.0.smithi188.stderr:++ termwidth= 2024-02-05T04:42:43.537 INFO:tasks.workunit.client.0.smithi188.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-02-05T04:42:43.537 INFO:tasks.workunit.client.0.smithi188.stderr:++ DIFFCOLOPTS='-y ' 2024-02-05T04:42:43.537 INFO:tasks.workunit.client.0.smithi188.stderr:++ KERNCORE=kernel.core_pattern 2024-02-05T04:42:43.537 INFO:tasks.workunit.client.0.smithi188.stderr:++ EXTRA_OPTS= 2024-02-05T04:42:43.538 INFO:tasks.workunit.client.0.smithi188.stderr:++ test '' = TESTS 2024-02-05T04:42:43.538 INFO:tasks.workunit.client.0.smithi188.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-02-05T04:42:43.538 INFO:tasks.workunit.client.0.smithi188.stderr:++ uname 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr:+ '[' Linux = FreeBSD ']' 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr:+ use_ec_overwrite=true 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr:+ getjson=no 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr:+ jqfilter='def walk(f): 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T04:42:43.540 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T04:42:43.541 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T04:42:43.542 INFO:tasks.workunit.client.0.smithi188.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T04:42:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:+ main osd-scrub-repair 2024-02-05T04:42:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:+ local dir=td/osd-scrub-repair 2024-02-05T04:42:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:+ shift 2024-02-05T04:42:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:+ shopt -s -o xtrace 2024-02-05T04:42:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T04:42:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:42:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-02-05T04:42:43.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-02-05T04:42:43.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-02-05T04:42:43.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none ' 2024-02-05T04:42:43.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-02-05T04:42:43.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-02-05T04:42:43.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T04:42:43.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-02-05T04:42:43.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_auto_repair_bluestore_basic 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_auto_repair_bluestore_failed 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_auto_repair_bluestore_scrub 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_auto_repair_bluestore_tag 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_auto_repair_erasure_coded_appends 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-02-05T04:42:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_and_repair_lrc_appends 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_and_repair_replicated 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_scrub_erasure_appends 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_scrub_replicated 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_corrupt_snapset_scrub_rep 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_list_missing_erasure_coded_appends 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_list_missing_erasure_coded_overwrites 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_periodic_scrub_replicated 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_repair_stats 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_repair_stats_ec 2024-02-05T04:42:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_request_scrub_priority 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_scrub_warning 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_skip_non_repair_during_recovery 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_unfound_erasure_coded_appends 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_unfound_erasure_coded_overwrites' 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:42:43.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:42:43.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:42:43.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:42:43.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:42:43.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:42:43.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:42:43.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:42:43.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:42:43.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:42:43.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:42:43.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:42:43.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:43.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:42:43.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:42:43.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:42:43.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:42:43.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:42:43.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:42:43.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:43.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:42:43.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:42:43.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:42:43.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T04:42:43.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T04:42:43.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T04:42:43.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T04:42:43.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:42:43.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:42:43.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:42:43.935 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:42:43.935 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:43.935 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:42:43.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:42:43.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T04:42:44.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:42:44.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:42:44.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:42:44.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:42:44.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:42:44.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:42:44.139 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:42:44.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T04:42:44.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:42:44.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T04:42:44.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-02-05T04:42:44.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T04:42:44.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:42:44.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:42:44.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:42:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T04:42:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:42:45.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:42:45.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:42:45.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:42:45.117 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:42:45.117 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:45.117 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:42:45.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:42:45.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:42:45.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:42:45.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T04:42:45.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:42:45.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:42:45.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:42:45.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:42:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:42:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T04:42:45.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:42:45.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:42:45.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee9889c1-7b34-41cc-9979-6dcf1a8b264a 2024-02-05T04:42:45.149 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 ee9889c1-7b34-41cc-9979-6dcf1a8b264a 2024-02-05T04:42:45.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ee9889c1-7b34-41cc-9979-6dcf1a8b264a' 2024-02-05T04:42:45.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:42:45.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFZ8BlZJrECRAAuwvExWIqdfVzXrtZjAi6FQ== 2024-02-05T04:42:45.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFZ8BlZJrECRAAuwvExWIqdfVzXrtZjAi6FQ=="}' 2024-02-05T04:42:45.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee9889c1-7b34-41cc-9979-6dcf1a8b264a -i td/osd-scrub-repair/0/new.json 2024-02-05T04:42:45.362 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:42:45.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T04:42:45.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBFZ8BlZJrECRAAuwvExWIqdfVzXrtZjAi6FQ== --osd-uuid ee9889c1-7b34-41cc-9979-6dcf1a8b264a 2024-02-05T04:42:45.404 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:45.403+0000 7fc25cca54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:42:45.415 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:45.413+0000 7fc25cca54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:42:45.416 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:45.414+0000 7fc25cca54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:42:45.416 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:45.414+0000 7fc25cca54c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:42:45.416 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:45.414+0000 7fc25cca54c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:42:45.416 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:45.414+0000 7fc25cca54c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:42:45.506 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:45.505+0000 7fc25cca54c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:42:50.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T04:42:50.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:42:50.881 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:42:50.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:42:50.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:42:51.251 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:42:51.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:42:51.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T04:42:51.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:42:51.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:42:51.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:42:51.279 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:51.278+0000 7efc04e0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:42:51.281 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:51.280+0000 7efc04e0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:42:51.282 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:51.281+0000 7efc04e0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:42:51.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:42:51.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:42:51.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:42:51.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:42:51.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:42:51.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:42:51.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:42:51.522 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:42:51.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:42:51.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:42:51.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:42:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:51.803+0000 7efc04e0f4c0 -1 Falling back to public interface 2024-02-05T04:42:52.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:42:52.789 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:42:52.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:42:52.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:42:52.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:42:52.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:42:53.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:42:54.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:42:54.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:42:54.063 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:42:54.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:42:54.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:42:54.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:42:54.122 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:42:54.122+0000 7efc04e0f4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:42:54.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:42:55.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:42:55.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:42:55.358 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:42:55.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:42:55.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:42:55.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:42:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:42:56.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:42:56.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:42:56.656 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:42:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:42:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:42:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:42:57.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:42:58.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:42:58.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:42:58.090 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:42:58.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:42:58.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:42:58.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:42:58.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:42:59.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:42:59.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:42:59.369 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:42:59.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:42:59.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:42:59.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:42:59.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:00.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:00.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:00.672 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:43:00.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:43:00.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:00.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:01.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:02.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:02.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:02.029 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:43:02.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:43:02.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:02.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:02.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:03.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:03.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:03.305 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:43:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:43:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:03.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:04.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:04.583 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:43:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:43:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:05.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:06.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:06.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:06.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:43:06.338 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:43:06.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:06.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:06.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:07.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:07.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:07.896 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:43:07.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:43:07.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:07.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:08.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:09.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:09.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:09.423 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:43:09.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:43:09.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:09.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:10.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:11.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:11.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:11.226 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:43:11.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:43:11.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:11.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:11.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:12.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:12.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:12.912 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:43:12.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:43:12.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:12.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:13.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:14.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:14.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:14.398 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:43:14.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:43:14.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:14.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:15.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:16.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:16.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:16.184 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:43:16.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:43:16.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:16.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:43:16.531 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2849389959,v1:127.0.0.1:6803/2849389959] [v2:127.0.0.1:6804/2849389959,v1:127.0.0.1:6805/2849389959] exists,up ee9889c1-7b34-41cc-9979-6dcf1a8b264a 2024-02-05T04:43:16.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:43:16.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:43:16.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:43:16.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:43:16.543 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:43:16.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3bc5fae-a921-4c3c-a982-4f61def17e78 2024-02-05T04:43:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c3bc5fae-a921-4c3c-a982-4f61def17e78' 2024-02-05T04:43:16.546 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 c3bc5fae-a921-4c3c-a982-4f61def17e78 2024-02-05T04:43:16.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:43:16.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkZ8BlfDTnIBAAEqGTBFTuJFkX5zvG63nzkw== 2024-02-05T04:43:16.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkZ8BlfDTnIBAAEqGTBFTuJFkX5zvG63nzkw=="}' 2024-02-05T04:43:16.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3bc5fae-a921-4c3c-a982-4f61def17e78 -i td/osd-scrub-repair/1/new.json 2024-02-05T04:43:16.904 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:43:16.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T04:43:16.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBkZ8BlfDTnIBAAEqGTBFTuJFkX5zvG63nzkw== --osd-uuid c3bc5fae-a921-4c3c-a982-4f61def17e78 2024-02-05T04:43:16.949 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:16.947+0000 7f1fc752b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:43:16.971 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:16.971+0000 7f1fc752b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:43:16.972 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:16.972+0000 7f1fc752b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:43:16.973 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:16.972+0000 7f1fc752b4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:43:16.973 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:16.972+0000 7f1fc752b4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:43:16.973 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:16.972+0000 7f1fc752b4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:43:17.038 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:17.037+0000 7f1fc752b4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:43:23.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T04:43:23.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:43:23.004 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:43:23.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:43:23.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:43:23.351 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:43:23.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:43:23.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T04:43:23.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:43:23.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:43:23.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:43:23.379 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:23.379+0000 7f2c8e0af4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:43:23.381 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:23.381+0000 7f2c8e0af4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:43:23.382 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:23.382+0000 7f2c8e0af4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:43:23.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:43:23.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:43:23.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:43:23.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:43:23.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:43:23.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:23.641 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:43:23.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:43:23.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:23.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:23.906+0000 7f2c8e0af4c0 -1 Falling back to public interface 2024-02-05T04:43:24.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:24.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:24.906 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:43:24.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:43:24.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:24.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:25.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:26.172 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:43:26.171+0000 7f2c8e0af4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:43:26.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:26.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:26.184 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:43:26.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:43:26.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:26.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:26.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:27.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:27.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:27.480 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:43:27.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:43:27.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:27.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:27.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:28.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:28.794 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:43:28.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:28.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:43:28.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:28.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:29.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:30.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:30.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:30.193 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:43:30.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:43:30.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:30.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:30.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:31.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:31.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:31.483 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:43:31.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:43:31.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:31.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:32.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:32.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:32.780 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:43:32.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:43:32.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:32.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:33.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:34.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:34.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:34.060 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:43:34.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:43:34.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:34.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:34.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:35.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:35.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:35.348 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:43:35.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:43:35.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:35.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:35.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:36.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:36.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:36.757 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:43:36.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:43:36.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:36.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:37.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:38.141 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:43:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:43:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:38.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:39.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:39.767 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:43:39.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:39.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:43:39.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:39.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:40.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:41.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:41.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:41.404 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:43:41.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:43:41.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:41.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:42.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:43.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:43.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:43.249 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:43:43.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:43:43.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:43.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:44.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:45.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:45.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:45.035 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:43:45.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:43:45.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:45.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:45.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:46.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:46.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:46.470 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:43:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:43:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:47.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:48.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:48.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:48.051 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:43:48.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:43:48.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:48.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:48.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:43:49.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:43:49.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:43:49.391 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:43:49.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:43:49.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:43:49.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:43:49.670 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3966624661,v1:127.0.0.1:6811/3966624661] [v2:127.0.0.1:6812/3966624661,v1:127.0.0.1:6813/3966624661] exists,up c3bc5fae-a921-4c3c-a982-4f61def17e78 2024-02-05T04:43:49.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:43:49.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:43:49.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:43:49.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-02-05T04:43:49.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T04:43:49.929 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T04:43:49.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T04:43:49.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T04:43:50.304 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T04:43:50.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:43:51.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T04:43:53.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-02-05T04:43:53.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:43:53.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:43:53.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:43:53.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:43:53.064 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:43:53.064 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:43:53.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:43:53.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:43:53.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:43:53.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:43:53.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:43:53.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:43:53.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:43:53.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:43:53.157 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:43:53.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:43:53.425 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T04:43:53.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:43:53.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:43:53.425 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:43:53.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-02-05T04:43:53.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-02-05T04:43:53.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-02-05T04:43:53.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:43:53.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:43:53.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-02-05T04:43:53.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-02-05T04:43:53.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672967' 2024-02-05T04:43:53.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:43:53.663 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-02-05T04:43:53.663 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:43:53.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:43:53.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-02-05T04:43:53.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:43:53.665 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836492 2024-02-05T04:43:53.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-02-05T04:43:53.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-02-05T04:43:53.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:43:53.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-02-05T04:43:53.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:43:54.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:43:54.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:43:55.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-02-05T04:43:55.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:43:55.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-02-05T04:43:55.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:43:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:43:55.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-02-05T04:43:55.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:43:55.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-02-05T04:43:55.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-02-05T04:43:55.229 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672967 2024-02-05T04:43:55.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:43:55.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-02-05T04:43:55.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:43:55.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:43:55.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:43:55.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:43:56.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T04:43:56.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:43:56.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:43:56.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:43:56.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T04:43:56.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T04:43:56.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T04:43:56.905 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T04:43:56.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T04:43:57.324 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T04:43:57.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T04:43:57.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T04:43:57.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T04:43:57.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-02-05T04:43:57.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-02-05T04:43:57.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:43:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-02-05T04:43:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T04:43:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:43:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T04:43:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:43:57.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:43:57.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T04:43:57.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:43:58.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-02-05T04:43:58.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:43:58.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:43:58.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:43:58.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:43:58.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:43:58.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:43:58.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:43:58.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:43:58.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:43:58.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:43:58.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:43:58.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:43:58.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:43:58.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T04:43:59.787 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:44:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:44:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:44:00.314 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:44:00.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:44:00.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:44:00.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:44:00.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:44:00.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:44:00.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:44:00.341 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:00.340+0000 7fe5e80144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:00.343 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:00.343+0000 7fe5e80144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:00.344 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:00.344+0000 7fe5e80144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:00.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:44:00.858 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:00.858+0000 7fe5e80144c0 -1 Falling back to public interface 2024-02-05T04:44:00.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:44:01.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:44:01.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:01.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:44:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:44:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:44:02.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:44:03.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:44:03.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:03.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:44:03.156 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:44:03.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:03.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:44:03.308 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:03.298+0000 7fe5e80144c0 -1 osd.0 22 log_to_monitors true 2024-02-05T04:44:03.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:44:04.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:44:04.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:44:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:44:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:44:04.857 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/2710817063,v1:127.0.0.1:6803/2710817063] [v2:127.0.0.1:6804/2710817063,v1:127.0.0.1:6805/2710817063] exists,up ee9889c1-7b34-41cc-9979-6dcf1a8b264a 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:44:04.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:44:04.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:44:04.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:44:04.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:44:04.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:44:04.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:44:04.939 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:44:05.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:44:05.223 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T04:44:05.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:44:05.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:44:05.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:44:05.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-02-05T04:44:05.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-02-05T04:44:05.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-02-05T04:44:05.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:44:05.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:44:05.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-02-05T04:44:05.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-02-05T04:44:05.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672971' 2024-02-05T04:44:05.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:44:05.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-02-05T04:44:05.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:44:05.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:44:05.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-02-05T04:44:05.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:44:05.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-02-05T04:44:05.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-02-05T04:44:05.443 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 111669149698 2024-02-05T04:44:05.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:44:05.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-02-05T04:44:05.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:44:06.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:44:06.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:44:06.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-02-05T04:44:06.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:44:07.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:44:07.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:44:08.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-02-05T04:44:08.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:44:08.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-02-05T04:44:08.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:44:08.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:44:08.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-02-05T04:44:08.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:44:08.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-02-05T04:44:08.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-02-05T04:44:08.260 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672971 2024-02-05T04:44:08.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:44:08.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-02-05T04:44:08.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:44:08.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:44:08.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:44:08.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T04:44:08.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:44:08.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:44:08.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:44:08.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:44:08.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:44:08.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:44:08.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:44:09.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T04:44:09.128 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:44:09.128 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:44:09.128 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:44:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T04:44:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:44:09.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:44:09.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-02-05T04:44:09.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T04:44:09.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T04:44:09.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T04:44:09.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T04:44:09.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-02-05T04:44:09.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-02-05T04:44:09.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-02-05T04:44:09.755 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-02-05T04:44:09.755 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:44:09.755 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:44:09.755 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:44:09.755 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:44:10.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:10.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-02-05T04:44:10.270 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.3 on osd.1 to repair 2024-02-05T04:44:10.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:10.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:44:10.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:44:10.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:43:50.248882+0000 '>' 2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:10.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:44:11.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:44:11.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:44:11.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T04:44:11.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:44:11.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:44:11.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:44:11.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:44:11.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:43:50.248882+0000 '>' 2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:11.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:44:12.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:44:12.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:44:12.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T04:44:12.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:44:12.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:44:12.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:44:12.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:44:13.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:43:50.248882+0000 '>' 2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:13.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:44:14.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:44:14.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:44:14.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T04:44:14.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:44:14.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:44:14.084 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:44:14.084 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:44:14.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:43:50.248882+0000 '>' 2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:14.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:44:15.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:44:15.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:44:15.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T04:44:15.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T04:44:15.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:44:15.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:44:15.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:44:11.105179+0000 '>' 2024-02-05T04:43:50.248882+0000 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:44:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:44:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:44:15.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T04:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T04:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T04:44:16.980 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T04:44:16.981 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T04:44:16.981 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T04:44:17.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T04:44:17.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:44:17.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:44:17.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:44:17.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:44:17.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:44:17.249 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:44:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:44:17.251 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T04:44:17.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:44:17.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T04:44:17.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:44:17.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:44:17.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:44:17.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:44:17.276 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:17.276+0000 7f59dad4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:17.298 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:17.298+0000 7f59dad4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:17.300 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:17.299+0000 7f59dad4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:17.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:17.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:44:17.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:44:17.815 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:17.815+0000 7f59dad4a4c0 -1 Falling back to public interface 2024-02-05T04:44:18.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:44:18.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:18.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:44:18.793 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:44:18.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:18.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:44:19.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:44:19.995 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:19.991+0000 7f59dad4a4c0 -1 osd.1 28 log_to_monitors true 2024-02-05T04:44:20.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:44:20.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:20.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:44:20.086 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:44:20.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:20.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:44:20.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:44:21.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:44:21.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:44:21.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:44:21.362 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:44:21.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:44:21.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:44:21.646 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/2632234616,v1:127.0.0.1:6811/2632234616] [v2:127.0.0.1:6812/2632234616,v1:127.0.0.1:6813/2632234616] exists,up c3bc5fae-a921-4c3c-a982-4f61def17e78 2024-02-05T04:44:21.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:44:21.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:44:21.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:44:21.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:44:21.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:44:21.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:44:21.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:44:21.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:44:21.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:44:21.994 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T04:44:21.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:44:21.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:44:21.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:44:22.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-02-05T04:44:22.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-02-05T04:44:22.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-02-05T04:44:22.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:44:22.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:44:22.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-02-05T04:44:22.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-02-05T04:44:22.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-02-05T04:44:22.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:44:22.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-02-05T04:44:22.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:44:22.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:44:22.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-02-05T04:44:22.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:44:22.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-02-05T04:44:22.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-02-05T04:44:22.203 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 111669149702 2024-02-05T04:44:22.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:44:22.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149702 2024-02-05T04:44:22.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:44:23.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:44:23.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:44:23.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-02-05T04:44:23.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:44:23.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-02-05T04:44:23.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:44:23.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:44:23.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-02-05T04:44:23.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:44:23.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-02-05T04:44:23.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-02-05T04:44:23.746 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 137438953474 2024-02-05T04:44:23.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:44:24.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-02-05T04:44:24.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:44:24.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:44:24.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:44:24.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:44:24.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T04:44:24.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:44:24.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:44:24.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:44:24.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T04:44:24.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:44:24.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:44:24.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T04:44:24.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T04:44:24.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T04:44:24.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:44:24.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:44:24.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:44:24.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:44:24.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:44:24.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:44:24.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:44:24.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:44:25.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:44:25.084 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:44:25.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:44:25.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:44:25.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:44:25.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:44:25.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:44:25.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:25.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:44:25.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:44:25.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:25.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:44:25.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:44:25.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:44:25.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:44:25.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:44:25.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:44:25.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:44:25.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:44:25.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:44:25.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:44:25.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:44:25.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:44:25.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:44:25.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:44:25.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:44:25.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:44:25.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:44:25.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:44:25.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:44:25.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:25.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:44:25.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:44:25.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:44:25.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:44:25.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:44:25.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:44:25.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:44:25.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:44:25.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:44:25.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:44:25.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:44:25.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:44:25.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:44:25.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:44:25.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:44:25.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:44:25.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:44:25.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:44:25.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:44:25.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:44:25.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-02-05T04:44:25.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-02-05T04:44:25.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-02-05T04:44:25.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-02-05T04:44:25.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-02-05T04:44:26.685 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 3fb4c610-1daa-4af4-840b-c160f805bf68 2024-02-05T04:44:26.940 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:44:26.983 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:26.982+0000 7f820c8be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:26.985 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:26.984+0000 7f820c8be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:26.986 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:26.985+0000 7f820c8be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:26.986+0000 7f820c8be4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:44:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:26.986+0000 7f820c8be4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:44:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:26.986+0000 7f820c8be4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:44:27.066 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:27.065+0000 7f820c8be4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:44:32.289 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:44:32.637 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:44:32.668 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:32.667+0000 7fe4482384c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:32.669 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:32.669+0000 7fe4482384c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:32.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:32.670+0000 7fe4482384c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:32.905 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:44:33.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:33.189+0000 7fe4482384c0 -1 Falling back to public interface 2024-02-05T04:44:34.170 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:44:35.443 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:44:35.480 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:35.479+0000 7fe4482384c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:44:36.756 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:44:38.035 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:44:39.340 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:44:40.620 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:44:41.900 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:44:43.179 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:44:44.472 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:44:45.752 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:44:47.060 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:44:48.479 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:44:49.806 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:44:51.261 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:44:52.753 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:44:54.733 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:44:56.125 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:44:57.863 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:44:58.139 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2429659944,v1:127.0.0.1:6803/2429659944] [v2:127.0.0.1:6804/2429659944,v1:127.0.0.1:6805/2429659944] exists,up 3fb4c610-1daa-4af4-840b-c160f805bf68 2024-02-05T04:44:58.142 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 a805f943-691c-4c2e-a7dd-53798464c536 2024-02-05T04:44:58.506 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:44:58.550 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:58.549+0000 7ff04180c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:58.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:58.582+0000 7ff04180c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:58.583 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:58.583+0000 7ff04180c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:44:58.584 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:58.583+0000 7ff04180c4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:44:58.584 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:58.583+0000 7ff04180c4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:44:58.584 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:58.583+0000 7ff04180c4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:44:58.660 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:44:58.659+0000 7ff04180c4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:45:04.277 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:45:04.621 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:45:04.651 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:04.650+0000 7f73466244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:04.653 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:04.652+0000 7f73466244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:04.654 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:04.653+0000 7f73466244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:04.891 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:45:05.177 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:05.177+0000 7f73466244c0 -1 Falling back to public interface 2024-02-05T04:45:06.159 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:45:07.433 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:45:07.508 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:07.508+0000 7f73466244c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:45:08.709 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:45:09.993 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:45:11.292 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:45:12.610 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:45:13.890 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:45:15.171 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:45:16.469 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:45:17.749 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:45:19.027 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:45:20.547 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:45:22.315 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:45:24.093 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:45:25.748 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:45:27.063 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:45:28.651 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:45:30.135 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:45:30.412 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/580290345,v1:127.0.0.1:6811/580290345] [v2:127.0.0.1:6812/580290345,v1:127.0.0.1:6813/580290345] exists,up a805f943-691c-4c2e-a7dd-53798464c536 2024-02-05T04:45:30.416 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 2fbb00e3-f473-418f-a73f-15a5d8768ec9 2024-02-05T04:45:30.790 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:45:30.834 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:30.834+0000 7f77f88254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:30.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:30.866+0000 7f77f88254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:30.867 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:30.866+0000 7f77f88254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:30.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:30.867+0000 7f77f88254c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:45:30.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:30.867+0000 7f77f88254c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:45:30.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:30.867+0000 7f77f88254c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:45:30.941 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:30.941+0000 7f77f88254c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T04:45:36.636 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:45:36.984 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:45:37.014 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:37.013+0000 7f4b762fa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:37.016 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:37.015+0000 7f4b762fa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:37.017 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:37.016+0000 7f4b762fa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:45:37.257 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:45:37.553 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:37.553+0000 7f4b762fa4c0 -1 Falling back to public interface 2024-02-05T04:45:38.523 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:45:39.831 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:45:39.884 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:45:39.883+0000 7f4b762fa4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:45:41.101 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:45:42.490 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:45:43.905 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:45:45.185 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:45:46.465 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:45:47.832 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:45:49.105 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:45:50.384 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:45:51.759 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:45:53.631 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:45:55.118 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:45:56.803 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:45:58.615 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:46:00.325 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:46:02.055 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:46:03.330 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:46:03.605 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2751194385,v1:127.0.0.1:6819/2751194385] [v2:127.0.0.1:6820/2751194385,v1:127.0.0.1:6821/2751194385] exists,up 2fbb00e3-f473-418f-a73f-15a5d8768ec9 2024-02-05T04:46:03.967 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'testpool' created 2024-02-05T04:46:05.765 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836498 2024-02-05T04:46:07.311 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672973 2024-02-05T04:46:07.580 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509444 2024-02-05T04:46:10.114 INFO:tasks.workunit.client.0.smithi188.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-02-05T04:46:10.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-02-05T04:46:10.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-02-05T04:46:10.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-02-05T04:46:10.467 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 2 2024-02-05T04:46:10.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-02-05T04:46:10.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:46:10.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:46:10.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:46:10.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:46:10.479 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:46:10.480 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:46:10.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:46:10.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:46:10.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:46:10.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:46:10.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:46:10.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:46:10.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:46:10.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:46:10.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:46:11.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:46:11.001 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:11.001 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:46:11.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:46:11.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:11.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:46:11.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T04:46:11.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T04:46:11.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T04:46:11.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:11.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:46:11.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T04:46:11.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T04:46:11.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-02-05T04:46:11.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:11.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:46:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-02-05T04:46:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-02-05T04:46:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509446' 2024-02-05T04:46:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:11.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T04:46:11.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:11.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:46:11.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T04:46:11.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:11.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T04:46:11.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T04:46:11.373 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836500 2024-02-05T04:46:11.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:11.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-02-05T04:46:11.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:46:12.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:46:12.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:12.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-02-05T04:46:12.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:12.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T04:46:12.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:12.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:46:12.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T04:46:12.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:12.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T04:46:12.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T04:46:12.919 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672976 2024-02-05T04:46:12.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:46:13.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T04:46:13.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:13.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-02-05T04:46:13.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:13.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:46:13.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-02-05T04:46:13.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:13.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-02-05T04:46:13.230 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509446 2024-02-05T04:46:13.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-02-05T04:46:13.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:46:13.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-02-05T04:46:13.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:46:13.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:13.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:13.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:46:13.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:46:13.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:46:13.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:46:13.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:46:13.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:46:13.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:46:13.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:46:14.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:46:14.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:46:14.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:14.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:14.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:46:14.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:46:14.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:46:14.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-02-05T04:46:14.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-02-05T04:46:14.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T04:46:14.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-02-05T04:46:14.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:46:14.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:46:14.558 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:46:14.558 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:46:14.558 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:46:14.558 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:14.558 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:46:14.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:46:14.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:14.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:46:15.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:46:15.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:46:15.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:46:15.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:46:15.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:46:15.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:46:15.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:46:15.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:46:15.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T04:46:16.650 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T04:46:17.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:46:17.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:17.171 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:46:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:46:17.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:46:17.173 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:46:17.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:46:17.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:46:17.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:46:17.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:46:17.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:46:17.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:46:17.198 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:17.198+0000 7f453b4114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:17.200+0000 7f453b4114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:17.202 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:17.201+0000 7f453b4114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:17.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:17.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:17.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:17.723 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:17.723+0000 7f453b4114c0 -1 Falling back to public interface 2024-02-05T04:46:18.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:18.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:18.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:46:18.713 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:18.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:18.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:19.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:20.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:20.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:20.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:46:20.026 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:46:20.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:20.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:20.057 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:20.056+0000 7f453b4114c0 -1 osd.0 21 log_to_monitors true 2024-02-05T04:46:20.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:21.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:21.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:21.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:46:21.303 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:46:21.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:21.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2851926392,v1:127.0.0.1:6803/2851926392] [v2:127.0.0.1:6804/2851926392,v1:127.0.0.1:6805/2851926392] exists,up 3fb4c610-1daa-4af4-840b-c160f805bf68 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:46:21.585 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:46:21.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:46:21.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:46:21.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:46:21.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:46:21.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:46:21.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:46:21.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:46:21.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:46:21.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:46:21.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:46:21.948 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:21.948 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:46:21.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:46:21.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:21.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:46:22.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T04:46:22.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T04:46:22.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T04:46:22.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:22.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:46:22.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-02-05T04:46:22.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-02-05T04:46:22.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672979' 2024-02-05T04:46:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:22.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:46:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T04:46:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T04:46:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672979 2-64424509449' 2024-02-05T04:46:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:22.257 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T04:46:22.257 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:46:22.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T04:46:22.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:22.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T04:46:22.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T04:46:22.259 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 107374182402 2024-02-05T04:46:22.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:22.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T04:46:22.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:46:23.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:46:23.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:23.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T04:46:23.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:23.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-02-05T04:46:23.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:23.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:46:23.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-02-05T04:46:23.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:23.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-02-05T04:46:23.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-02-05T04:46:23.801 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672979 2024-02-05T04:46:23.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:46:24.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-02-05T04:46:24.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:24.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T04:46:24.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:24.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:46:24.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T04:46:24.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:24.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T04:46:24.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T04:46:24.071 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509449 2024-02-05T04:46:24.071 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:46:24.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T04:46:24.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:46:24.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:24.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:46:24.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:46:24.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:46:24.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:46:24.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:24.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:25.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:46:25.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:46:25.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:46:25.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-02-05T04:46:25.416 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T04:46:25.416 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T04:46:25.416 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:46:25.430 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T04:46:25.430 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-02-05T04:46:25.430 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:46:25.444 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2: { 2024-02-05T04:46:25.444 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-02-05T04:46:25.444 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:46:25.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-02-05T04:46:25.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T04:46:25.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T04:46:25.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:25.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T04:46:25.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-02-05T04:46:25.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-02-05T04:46:25.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:46:25.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:46:25.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:25.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:46:26.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-02-05T04:46:26.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-02-05T04:46:26.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:46:26.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:46:26.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:46:26.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:46:26.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:26.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-02-05T04:46:26.361 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:46:26.362 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T04:46:26.362 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T04:46:26.362 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-22T04:44:46.361890+0000" 2024-02-05T04:46:26.362 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:46:26.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:26.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T04:46:26.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:26.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T04:46:26.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T04:46:26.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:46:26.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:46:26.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:46:26.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:46:26.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:46:26.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:46:26.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:46:03.917416+0000 '>' 2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:26.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:46:27.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:46:27.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:46:27.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:46:27.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:46:27.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:46:27.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:46:27.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:46:27.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:46:03.917416+0000 '>' 2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:27.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:46:28.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:46:28.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:46:28.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:46:28.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:46:28.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:46:28.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:46:28.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:46:29.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:46:03.917416+0000 '>' 2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:29.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:46:30.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:46:30.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:46:30.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:46:30.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:46:30.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:46:30.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:46:30.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:46:30.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:46:03.917416+0000 '>' 2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:30.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:46:31.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:46:31.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:46:31.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:46:31.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:46:31.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:46:31.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:46:31.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:46:31.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:46:03.917416+0000 '>' 2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:31.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:46:32.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:46:32.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:46:32.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:46:32.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:46:32.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:46:32.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:46:32.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:46:27.092671+0000 '>' 2024-02-05T04:46:03.917416+0000 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:46:33.038 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:46:33.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:46:33.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:46:33.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:46:33.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:46:33.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:46:33.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:46:33.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:46:33.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:46:33.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:46:33.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:46:33.386 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:33.386 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:46:33.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:46:33.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:33.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:46:33.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-02-05T04:46:33.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-02-05T04:46:33.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-02-05T04:46:33.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:33.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:46:33.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-02-05T04:46:33.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-02-05T04:46:33.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672982' 2024-02-05T04:46:33.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:33.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:46:33.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-05T04:46:33.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-05T04:46:33.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672982 2-64424509453' 2024-02-05T04:46:33.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:33.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-02-05T04:46:33.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:33.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:46:33.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-02-05T04:46:33.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:33.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-02-05T04:46:33.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-02-05T04:46:33.698 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 107374182405 2024-02-05T04:46:33.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:33.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-02-05T04:46:33.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:46:34.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:46:34.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-02-05T04:46:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:35.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-02-05T04:46:35.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:46:35.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-02-05T04:46:35.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-02-05T04:46:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-02-05T04:46:35.246 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672982 2024-02-05T04:46:35.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:46:35.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-02-05T04:46:35.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:35.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-05T04:46:35.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:35.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:46:35.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-05T04:46:35.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:35.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-05T04:46:35.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-05T04:46:35.518 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509453 2024-02-05T04:46:35.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:46:35.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-02-05T04:46:35.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:46:35.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:35.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:36.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:46:36.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:46:36.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:46:36.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:46:36.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:46:36.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:46:36.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:46:36.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:46:36.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:46:36.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:46:36.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:36.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:36.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:46:36.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:46:36.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:46:36.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-02-05T04:46:37.020 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:46:37.020 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T04:46:27.244095+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-02-05T04:46:27.092671+0000 21'1 2024-02-05T04:46:27.092671+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:46:27.092671+0000 1 0 2024-02-05T04:46:37.020 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:46:37.020 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:46:37.021 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:37.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:46:37.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:46:37.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:37.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:46:37.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:46:37.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:46:37.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:46:37.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:46:37.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:46:37.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:46:37.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:46:37.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:46:37.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:46:37.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:46:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:46:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:46:37.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T04:46:39.131 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T04:46:39.131 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T04:46:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T04:46:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:46:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:46:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:46:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:46:39.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:46:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:46:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:46:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:46:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:46:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:46:39.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:46:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:46:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:46:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:46:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:46:39.433 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:39.432+0000 7fe74ab744c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:39.435 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:39.434+0000 7fe74ab744c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:39.436 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:39.435+0000 7fe74ab744c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:39.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:46:39.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:39.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:39.969 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:39.969+0000 7fe74ab744c0 -1 Falling back to public interface 2024-02-05T04:46:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:46:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:41.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:42.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:42.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:42.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:46:42.225 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:46:42.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:42.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:42.235 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:42.235+0000 7fe74ab744c0 -1 osd.0 26 log_to_monitors true 2024-02-05T04:46:42.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:46:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:46:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/1627566033,v1:127.0.0.1:6803/1627566033] [v2:127.0.0.1:6804/1627566033,v1:127.0.0.1:6805/1627566033] exists,up 3fb4c610-1daa-4af4-840b-c160f805bf68 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:46:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:46:43.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:46:43.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:46:43.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:46:43.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:46:43.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:46:43.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:46:43.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:46:43.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:46:43.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:46:44.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:46:44.164 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:44.164 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:46:44.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:46:44.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:44.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:46:44.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T04:46:44.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T04:46:44.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-02-05T04:46:44.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:44.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:46:44.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-02-05T04:46:44.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-02-05T04:46:44.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985' 2024-02-05T04:46:44.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:44.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:46:44.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-02-05T04:46:44.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-02-05T04:46:44.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672985 2-64424509456' 2024-02-05T04:46:44.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:44.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-02-05T04:46:44.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:44.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:46:44.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-02-05T04:46:44.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:44.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T04:46:44.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-02-05T04:46:44.477 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 128849018882 2024-02-05T04:46:44.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:44.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T04:46:44.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:44.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-02-05T04:46:44.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:44.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:46:44.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-02-05T04:46:44.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:44.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-02-05T04:46:44.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-02-05T04:46:44.796 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672985 2024-02-05T04:46:44.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:46:45.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-02-05T04:46:45.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:45.098 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-02-05T04:46:45.098 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:45.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:46:45.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-02-05T04:46:45.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:45.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-02-05T04:46:45.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-02-05T04:46:45.101 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509456 2024-02-05T04:46:45.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:46:45.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-02-05T04:46:45.368 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:46:45.368 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:45.368 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:45.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:46:45.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:46:45.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:46:45.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:46:45.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:46:45.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:46:45.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:46:45.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:46:45.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:46:45.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:46:45.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:45.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:46:46.295 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:46:46.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:46:46.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:46.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:46:46.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:46:46.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:46:46.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:46:46.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:46:46.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:46:46.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:46:46.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:46:46.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:46:47.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:46:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:46:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:46:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:46:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:46:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:46:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:46:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:46:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:46:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:46:48.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T04:46:48.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:46:48.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:46:48.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:46:48.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:46:48.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:46:48.680 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:48.680+0000 7fa27bd8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:48.683 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:48.682+0000 7fa27bd8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:48.684 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:48.683+0000 7fa27bd8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:48.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:48.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:49.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:49.203+0000 7fa27bd8d4c0 -1 Falling back to public interface 2024-02-05T04:46:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:46:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:51.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:51.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:51.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:46:51.479 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:46:51.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:51.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:51.499 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:51.498+0000 7fa27bd8d4c0 -1 osd.0 31 log_to_monitors true 2024-02-05T04:46:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:46:52.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:46:52.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:46:52.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:46:52.804 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:46:52.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:46:52.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:46:53.141 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/3761053315,v1:127.0.0.1:6803/3761053315] [v2:127.0.0.1:6804/3761053315,v1:127.0.0.1:6805/3761053315] exists,up 3fb4c610-1daa-4af4-840b-c160f805bf68 2024-02-05T04:46:53.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:46:53.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:46:53.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:46:53.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:46:53.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:46:53.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:46:53.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:46:53.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:46:53.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:46:53.495 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:46:53.496 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:46:53.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:46:53.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:53.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:46:53.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-02-05T04:46:53.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-02-05T04:46:53.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-02-05T04:46:53.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:53.599 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:46:53.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-02-05T04:46:53.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-02-05T04:46:53.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672988' 2024-02-05T04:46:53.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:46:53.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:46:53.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T04:46:53.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T04:46:53.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672988 2-64424509459' 2024-02-05T04:46:53.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:53.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-02-05T04:46:53.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:53.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:46:53.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-02-05T04:46:53.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:53.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-02-05T04:46:53.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-02-05T04:46:53.808 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 150323855362 2024-02-05T04:46:53.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:54.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-02-05T04:46:54.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:46:55.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:46:55.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:46:55.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-02-05T04:46:55.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:55.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-02-05T04:46:55.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:55.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:46:55.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-02-05T04:46:55.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:55.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-02-05T04:46:55.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-02-05T04:46:55.352 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672988 2024-02-05T04:46:55.352 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:46:55.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-02-05T04:46:55.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:46:55.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T04:46:55.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:46:55.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:46:55.620 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T04:46:55.620 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:46:55.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T04:46:55.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T04:46:55.622 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509459 2024-02-05T04:46:55.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:46:55.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-02-05T04:46:55.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:46:55.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:55.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:46:56.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:46:56.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:46:56.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:46:56.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:46:56.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:46:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:46:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:46:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:46:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T04:46:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T04:46:56.877 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:46:27.091+0000 7f7329f4b640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:46:56.877 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:46:27.091+0000 7f7329f4b640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T04:46:56.877 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:46:27.091+0000 7f7329f4b640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T04:46:56.877 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:46:27.091+0000 7f7329f4b640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-02-05T04:46:56.877 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:46:27.091+0000 7f7329f4b640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:46:56.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:46:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:46:56.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:46:56.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:46:56.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:46:56.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:46:56.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:46:56.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:46:56.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:46:56.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:46:56.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:46:56.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:46:56.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:46:56.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:46:56.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:46:56.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:46:57.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:46:57.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:57.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:57.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:46:57.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:46:57.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:46:57.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:46:57.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:46:57.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:46:57.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:46:57.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:46:57.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:46:57.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:46:57.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:46:57.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:46:57.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:46:57.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:46:57.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:46:57.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:46:57.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:46:57.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:46:57.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:57.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:57.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:46:57.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:46:57.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:46:57.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:46:57.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:46:57.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:57.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:57.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-02-05T04:46:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-02-05T04:46:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T04:46:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:46:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:46:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:46:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T04:46:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T04:46:57.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:46:57.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:46:57.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:46:57.382 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:46:57.382 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:57.383 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:57.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:57.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T04:46:57.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:46:57.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:46:57.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:46:57.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:46:57.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:46:57.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:46:57.593 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:46:57.593 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:46:57.594 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:46:57.594 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:46:57.594 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:57.594 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:57.594 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:46:57.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:46:57.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T04:46:57.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:46:57.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:46:57.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:46:57.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T04:46:57.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-02-05T04:46:57.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T04:46:57.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:46:57.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:46:57.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:46:57.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T04:46:57.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:46:58.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:46:58.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:46:58.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:46:58.603 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:46:58.603 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:58.603 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:58.603 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:58.603 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:46:58.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:46:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:46:58.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:46:58.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:46:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:46:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:46:58.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:46:58.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ff8a9b9a-1dc9-4791-872d-d14529e2e327 2024-02-05T04:46:58.638 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 ff8a9b9a-1dc9-4791-872d-d14529e2e327 2024-02-05T04:46:58.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ff8a9b9a-1dc9-4791-872d-d14529e2e327' 2024-02-05T04:46:58.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:46:58.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCaMBlfyfgJhAAXGHTlouu3puo+PkCPgFXcA== 2024-02-05T04:46:58.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCaMBlfyfgJhAAXGHTlouu3puo+PkCPgFXcA=="}' 2024-02-05T04:46:58.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ff8a9b9a-1dc9-4791-872d-d14529e2e327 -i td/osd-scrub-repair/0/new.json 2024-02-05T04:46:58.859 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:46:58.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T04:46:58.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBCaMBlfyfgJhAAXGHTlouu3puo+PkCPgFXcA== --osd-uuid ff8a9b9a-1dc9-4791-872d-d14529e2e327 2024-02-05T04:46:58.900 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:58.899+0000 7f4c6c80d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:58.909 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:58.908+0000 7f4c6c80d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:58.910 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:58.909+0000 7f4c6c80d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:46:58.911 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:58.910+0000 7f4c6c80d4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:46:58.911 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:58.910+0000 7f4c6c80d4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:46:58.911 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:58.910+0000 7f4c6c80d4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:46:59.001 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:46:59.000+0000 7f4c6c80d4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:47:04.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T04:47:04.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:47:04.525 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:47:04.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:47:04.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:47:04.869 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:47:04.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:47:04.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:47:04.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:47:04.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:47:04.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:47:04.897 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:04.896+0000 7f3657b074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:04.899 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:04.898+0000 7f3657b074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:04.900 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:04.899+0000 7f3657b074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:05.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:47:05.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:47:05.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:47:05.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:47:05.139 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:47:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:47:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:47:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:05.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:05.431 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:05.431+0000 7f3657b074c0 -1 Falling back to public interface 2024-02-05T04:47:06.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:06.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:06.410 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:47:06.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:47:06.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:06.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:06.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:07.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:07.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:07.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:47:07.687 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:47:07.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:07.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:07.827 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:07.826+0000 7f3657b074c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:47:08.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:09.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:09.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:09.020 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:47:09.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:47:09.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:09.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:09.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:10.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:10.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:10.300 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:47:10.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:47:10.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:10.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:10.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:11.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:11.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:11.658 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:47:11.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:47:11.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:11.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:11.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:12.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:12.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:12.935 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:47:12.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:47:12.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:12.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:13.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:14.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:14.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:14.214 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:47:14.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:47:14.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:14.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:14.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:15.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:15.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:15.497 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:47:15.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:47:15.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:15.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:15.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:16.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:16.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:16.828 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:47:16.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:47:16.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:16.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:17.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:18.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:18.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:18.108 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:47:18.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:47:18.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:18.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:18.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:19.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:19.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:19.607 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:47:19.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:47:19.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:19.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:20.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:21.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:21.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:21.068 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:47:21.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:47:21.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:21.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:21.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:22.441 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:47:22.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:47:22.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:22.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:23.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:24.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:24.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:24.222 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:47:24.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:47:24.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:24.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:24.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:25.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:25.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:25.610 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:47:25.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:47:25.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:25.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:26.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:27.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:27.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:27.062 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:47:27.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:47:27.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:27.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:27.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:28.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:28.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:28.755 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:47:28.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:47:28.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:28.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:29.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:30.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:30.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:30.049 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:47:30.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:47:30.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:30.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:47:30.323 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4145330314,v1:127.0.0.1:6803/4145330314] [v2:127.0.0.1:6804/4145330314,v1:127.0.0.1:6805/4145330314] exists,up ff8a9b9a-1dc9-4791-872d-d14529e2e327 2024-02-05T04:47:30.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:47:30.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:47:30.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:47:30.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13e5c017-87c5-4796-848d-b3d08e7401a3 2024-02-05T04:47:30.331 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 13e5c017-87c5-4796-848d-b3d08e7401a3 2024-02-05T04:47:30.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 13e5c017-87c5-4796-848d-b3d08e7401a3' 2024-02-05T04:47:30.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:47:30.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiaMBl0q6MFBAArf15x/suA9ydNmbk45oc7g== 2024-02-05T04:47:30.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiaMBl0q6MFBAArf15x/suA9ydNmbk45oc7g=="}' 2024-02-05T04:47:30.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13e5c017-87c5-4796-848d-b3d08e7401a3 -i td/osd-scrub-repair/1/new.json 2024-02-05T04:47:30.710 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:47:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T04:47:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBiaMBl0q6MFBAArf15x/suA9ydNmbk45oc7g== --osd-uuid 13e5c017-87c5-4796-848d-b3d08e7401a3 2024-02-05T04:47:30.751 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:30.749+0000 7fa511a124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:30.795 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:30.793+0000 7fa511a124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:30.796 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:30.794+0000 7fa511a124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:30.796 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:30.794+0000 7fa511a124c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:47:30.796 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:30.794+0000 7fa511a124c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:47:30.796 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:30.794+0000 7fa511a124c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:47:30.870 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:30.868+0000 7fa511a124c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:47:36.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T04:47:36.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:47:36.338 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:47:36.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:47:36.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:47:36.686 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:47:36.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:47:36.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:47:36.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:47:36.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:47:36.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:47:36.714 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:36.713+0000 7f1d5a5074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:36.736 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:36.735+0000 7f1d5a5074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:36.737 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:36.736+0000 7f1d5a5074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:47:36.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:47:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:47:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:47:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:47:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:47:36.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:36.957 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:47:36.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:47:36.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:36.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:37.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:37.252 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:37.252+0000 7f1d5a5074c0 -1 Falling back to public interface 2024-02-05T04:47:38.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:38.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:38.224 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:47:38.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:47:38.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:38.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:38.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:39.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:39.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:39.493 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:47:39.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:47:39.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:39.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:39.578 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:47:39.577+0000 7f1d5a5074c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:47:39.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:40.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:40.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:40.788 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:47:40.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:47:40.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:40.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:41.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:42.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:42.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:42.194 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:47:42.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:47:42.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:42.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:42.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:43.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:43.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:43.575 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:47:43.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:47:43.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:43.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:43.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:44.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:44.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:44.852 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:47:44.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:47:44.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:44.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:45.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:46.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:46.123 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:47:46.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:46.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:47:46.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:46.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:46.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:47.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:47.399 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:47:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:47:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:47.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:48.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:48.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:48.677 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:47:48.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:47:48.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:48.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:48.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:50.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:50.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:50.000 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:47:50.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:47:50.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:50.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:50.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:51.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:51.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:51.357 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:47:51.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:47:51.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:51.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:52.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:53.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:53.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:53.045 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:47:53.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:47:53.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:53.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:53.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:47:54.337 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:47:54.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:54.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:54.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:55.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:55.748 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:47:55.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:55.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:47:55.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:55.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:56.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:57.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:57.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:57.509 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:47:57.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:47:57.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:57.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:47:58.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:47:59.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:47:59.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:47:59.287 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:47:59.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:47:59.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:47:59.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:48:00.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:01.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:01.042 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:48:01.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:01.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:48:01.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:01.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:48:01.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:02.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:02.943 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:48:02.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:02.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:48:02.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:02.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:48:03.220 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63136612,v1:127.0.0.1:6811/63136612] [v2:127.0.0.1:6812/63136612,v1:127.0.0.1:6813/63136612] exists,up 13e5c017-87c5-4796-848d-b3d08e7401a3 2024-02-05T04:48:03.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:48:03.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:48:03.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:48:03.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-02-05T04:48:03.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:48:03.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:48:03.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:48:03.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:48:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:48:03.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:48:03.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:48:03.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T04:48:03.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:48:03.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=32dd84e7-f927-42da-bd15-e86eddd056d0 2024-02-05T04:48:03.227 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 32dd84e7-f927-42da-bd15-e86eddd056d0 2024-02-05T04:48:03.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 32dd84e7-f927-42da-bd15-e86eddd056d0' 2024-02-05T04:48:03.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:48:03.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDaMBlUSlTDhAAVoOtqXbUcT3yZKzWqzQQOQ== 2024-02-05T04:48:03.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDaMBlUSlTDhAAVoOtqXbUcT3yZKzWqzQQOQ=="}' 2024-02-05T04:48:03.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 32dd84e7-f927-42da-bd15-e86eddd056d0 -i td/osd-scrub-repair/2/new.json 2024-02-05T04:48:03.604 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:48:03.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T04:48:03.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCDaMBlUSlTDhAAVoOtqXbUcT3yZKzWqzQQOQ== --osd-uuid 32dd84e7-f927-42da-bd15-e86eddd056d0 2024-02-05T04:48:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:03.645+0000 7ff1c87944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:03.688 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:03.688+0000 7ff1c87944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:03.690 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:03.689+0000 7ff1c87944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:03.690 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:03.689+0000 7ff1c87944c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:48:03.690 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:03.689+0000 7ff1c87944c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:48:03.690 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:03.689+0000 7ff1c87944c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:48:03.772 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:03.771+0000 7ff1c87944c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T04:48:09.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T04:48:09.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:48:09.606 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:48:09.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:48:09.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:48:09.949 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:48:09.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:48:09.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:48:09.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:48:09.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:48:09.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:48:09.978 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:09.977+0000 7fbc8feb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:09.979 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:09.979+0000 7fbc8feb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:09.980 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:09.980+0000 7fbc8feb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:10.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:48:10.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:48:10.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:48:10.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:48:10.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:48:10.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:10.217 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:48:10.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:48:10.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:10.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:10.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:10.504 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:10.504+0000 7fbc8feb44c0 -1 Falling back to public interface 2024-02-05T04:48:11.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:11.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:11.484 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:48:11.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:48:11.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:11.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:11.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:12.760 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:48:12.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:12.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:48:12.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:12.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:12.971 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:12.970+0000 7fbc8feb44c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:48:13.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:14.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:14.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:14.164 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:48:14.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:48:14.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:14.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:14.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:15.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:15.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:15.443 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:48:15.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:48:15.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:15.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:15.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:16.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:16.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:16.767 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:48:16.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:48:16.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:16.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:17.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:18.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:18.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:18.074 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:48:18.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:48:18.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:18.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:18.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:19.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:19.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:19.351 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:48:19.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:48:19.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:19.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:19.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:20.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:20.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:20.726 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:48:20.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:48:20.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:20.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:21.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:22.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:22.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:22.074 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:48:22.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:48:22.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:22.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:22.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:23.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:23.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:23.354 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:48:23.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:48:23.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:23.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:23.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:24.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:24.698 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:48:24.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:24.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:48:24.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:24.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:25.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:26.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:26.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:26.525 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:48:26.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:48:26.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:26.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:27.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:28.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:28.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:28.114 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:48:28.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:48:28.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:28.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:28.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:29.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:29.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:29.861 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:48:29.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:48:29.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:29.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:30.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:31.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:31.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:31.363 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:48:31.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:48:31.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:31.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:31.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:32.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:32.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:32.822 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:48:32.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:48:32.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:32.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:33.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:34.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:34.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:34.683 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:48:34.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:48:34.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:34.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:35.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:36.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:36.462 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:48:36.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:36.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:48:36.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:36.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:48:36.735 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/142462473,v1:127.0.0.1:6819/142462473] [v2:127.0.0.1:6820/142462473,v1:127.0.0.1:6821/142462473] exists,up 32dd84e7-f927-42da-bd15-e86eddd056d0 2024-02-05T04:48:36.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:48:36.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:48:36.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:48:36.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-02-05T04:48:36.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T04:48:37.098 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'testpool' created 2024-02-05T04:48:37.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:48:38.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-02-05T04:48:38.477 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 2 2024-02-05T04:48:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-02-05T04:48:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:48:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:48:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:48:38.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:48:38.491 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:48:38.491 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:48:38.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:48:38.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:48:38.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:48:38.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:48:38.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:48:38.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:48:38.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:48:38.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:48:38.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:48:38.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:48:38.958 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:48:38.959 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:48:38.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:48:38.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:48:38.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:48:39.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-02-05T04:48:39.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-02-05T04:48:39.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-02-05T04:48:39.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:48:39.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:48:39.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T04:48:39.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T04:48:39.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-02-05T04:48:39.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:48:39.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:48:39.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-02-05T04:48:39.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-02-05T04:48:39.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509448' 2024-02-05T04:48:39.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:48:39.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-02-05T04:48:39.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:48:39.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:48:39.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-02-05T04:48:39.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:48:39.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-02-05T04:48:39.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-02-05T04:48:39.306 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836498 2024-02-05T04:48:39.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:48:39.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-02-05T04:48:39.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:48:40.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:48:40.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:48:40.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836498 2024-02-05T04:48:40.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:48:40.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T04:48:40.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:48:40.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:48:40.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T04:48:40.849 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:48:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T04:48:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T04:48:40.850 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672976 2024-02-05T04:48:40.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:48:41.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T04:48:41.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:48:41.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-02-05T04:48:41.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:48:41.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:48:41.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-02-05T04:48:41.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:48:41.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-02-05T04:48:41.142 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509448 2024-02-05T04:48:41.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-02-05T04:48:41.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:48:41.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-02-05T04:48:41.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:48:41.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:48:41.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:48:41.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:48:41.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:48:41.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:48:41.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:48:41.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:48:41.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:48:41.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:48:41.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:48:42.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:48:42.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:48:42.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:48:42.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:48:42.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:48:42.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T04:48:42.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T04:48:42.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T04:48:42.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:48:42.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T04:48:42.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:48:42.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:48:42.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:48:42.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:48:42.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:48:42.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:48:42.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:48:42.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:48:42.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:48:42.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:48:42.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:48:42.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:48:43.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:48:43.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T04:48:43.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T04:48:43.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T04:48:43.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T04:48:43.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T04:48:43.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T04:48:43.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T04:48:43.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T04:48:43.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T04:48:43.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T04:48:43.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T04:48:43.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:48:43.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T04:48:43.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:48:43.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:48:43.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:48:43.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:48:43.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:48:43.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:48:43.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:48:43.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:48:43.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:48:43.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:48:43.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:48:43.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T04:48:44.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T04:48:44.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T04:48:44.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T04:48:44.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T04:48:44.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:48:44.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T04:48:44.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:48:44.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:48:44.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:48:44.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:48:44.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:48:44.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:48:44.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:48:44.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:48:45.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:48:45.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:48:45.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:48:45.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:48:45.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:48:45.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:48:45.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:48:45.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-02-05T04:48:45.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-02-05T04:48:45.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-02-05T04:48:45.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:45.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:45.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:45.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:45.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:45.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:45.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:45.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:46.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:46.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:46.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:46.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:46.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:46.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:46.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:46.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:46.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:46.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:46.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-02-05T04:48:46.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:48:46.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-02-05T04:48:46.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:48:46.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:48:46.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:48:46.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:48:46.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:48:46.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:48:46.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:48:47.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:48:47.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:48:47.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:48:47.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-02-05T04:48:47.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:48:47.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:48:47.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:48:47.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-02-05T04:48:49.151 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T04:48:49.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:48:49.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:48:49.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:48:49.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:48:49.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:48:49.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:48:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:48:49.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:48:49.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:48:49.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:48:49.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:48:49.680 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:48:49.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:48:49.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:48:49.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:48:49.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:48:49.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:48:49.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:48:49.707 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:49.707+0000 7f0ee39174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:49.709 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:49.709+0000 7f0ee39174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:49.711 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:49.710+0000 7f0ee39174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:48:49.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:48:49.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:49.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:48:50.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:50.235 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:50.234+0000 7f0ee39174c0 -1 Falling back to public interface 2024-02-05T04:48:51.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:51.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:51.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:48:51.220 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:48:51.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:51.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:48:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:52.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:52.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:52.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:48:52.496 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:48:52.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:52.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:48:52.612 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:48:52.611+0000 7f0ee39174c0 -1 osd.0 21 log_to_monitors true 2024-02-05T04:48:52.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:48:53.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:48:53.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:48:53.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:48:53.774 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:48:53.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:48:53.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3364301451,v1:127.0.0.1:6803/3364301451] [v2:127.0.0.1:6804/3364301451,v1:127.0.0.1:6805/3364301451] exists,up ff8a9b9a-1dc9-4791-872d-d14529e2e327 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:48:54.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:48:54.078 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:48:54.078 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:48:54.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:48:54.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:48:54.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:48:54.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:48:54.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:48:54.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:48:54.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:48:54.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:48:54.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:48:54.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:48:54.479 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:48:54.479 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:48:54.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:48:54.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:48:54.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:48:54.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T04:48:54.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T04:48:54.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T04:48:54.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:48:54.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:48:54.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-02-05T04:48:54.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-02-05T04:48:54.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980' 2024-02-05T04:48:54.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:48:54.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:48:54.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T04:48:54.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T04:48:54.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672980 2-64424509452' 2024-02-05T04:48:54.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:48:54.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T04:48:54.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:48:54.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:48:54.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T04:48:54.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:48:54.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T04:48:54.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T04:48:54.804 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 107374182402 2024-02-05T04:48:54.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:48:55.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T04:48:55.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:48:56.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:48:56.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:48:56.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T04:48:56.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:48:57.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:48:57.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:48:57.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T04:48:57.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:48:57.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-02-05T04:48:57.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:48:57.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:48:57.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-02-05T04:48:57.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:48:57.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-02-05T04:48:57.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-02-05T04:48:57.628 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672980 2024-02-05T04:48:57.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:48:57.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-02-05T04:48:57.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:48:57.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T04:48:57.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:48:57.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:48:57.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T04:48:57.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:48:57.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T04:48:57.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T04:48:57.899 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509452 2024-02-05T04:48:57.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:48:58.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-02-05T04:48:58.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:48:58.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:48:58.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:48:58.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:48:58.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:48:58.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:48:58.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:48:58.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:48:59.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:48:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:48:59.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:48:59.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:48:59.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:48:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:48:59.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:48:59.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:48:59.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:48:59.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:48:59.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:48:59.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:48:59.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:48:59.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T04:48:59.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:48:59.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:48:59.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:48:59.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:48:59.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:48:59.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T04:49:01.245 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:49:01.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:49:01.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:49:01.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:49:01.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:49:01.766 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:49:01.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:01.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:49:01.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:49:01.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:49:01.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:49:01.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:49:01.794 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:01.794+0000 7f7d4620e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:01.796 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:01.796+0000 7f7d4620e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:01.797 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:01.797+0000 7f7d4620e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:02.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:02.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:02.319 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:02.319+0000 7f7d4620e4c0 -1 Falling back to public interface 2024-02-05T04:49:03.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:49:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:03.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:03.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:49:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:49:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:04.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:04.597 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:04.597+0000 7f7d4620e4c0 -1 osd.0 26 log_to_monitors true 2024-02-05T04:49:04.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:05.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:05.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:05.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:49:05.870 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:49:05.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:05.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:06.146 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3279281968,v1:127.0.0.1:6803/3279281968] [v2:127.0.0.1:6804/3279281968,v1:127.0.0.1:6805/3279281968] exists,up ff8a9b9a-1dc9-4791-872d-d14529e2e327 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:49:06.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:49:06.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:49:06.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:49:06.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:49:06.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:49:06.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:49:06.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:49:06.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:49:06.537 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:06.537 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:49:06.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:49:06.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:06.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:49:06.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T04:49:06.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T04:49:06.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-02-05T04:49:06.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:06.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:49:06.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-02-05T04:49:06.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-02-05T04:49:06.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983' 2024-02-05T04:49:06.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:06.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:49:06.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-02-05T04:49:06.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-02-05T04:49:06.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983 2-64424509456' 2024-02-05T04:49:06.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:06.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-02-05T04:49:06.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:06.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:49:06.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-02-05T04:49:06.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:06.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T04:49:06.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-02-05T04:49:06.853 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 128849018882 2024-02-05T04:49:06.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:07.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-02-05T04:49:07.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:08.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:49:08.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:08.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-02-05T04:49:08.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:09.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:49:09.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:09.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T04:49:09.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:09.675 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-02-05T04:49:09.675 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:09.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:49:09.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-02-05T04:49:09.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-02-05T04:49:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-02-05T04:49:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672983 2024-02-05T04:49:09.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:09.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672983 2024-02-05T04:49:09.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:09.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-02-05T04:49:09.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:09.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:49:09.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-02-05T04:49:09.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:09.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-02-05T04:49:09.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-02-05T04:49:09.948 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509456 2024-02-05T04:49:09.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:49:10.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-02-05T04:49:10.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:49:10.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:10.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:10.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:49:10.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:49:10.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:49:10.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:49:10.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:49:10.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:49:10.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:49:10.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:49:10.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:49:10.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:49:10.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:10.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:49:11.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:49:11.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T04:49:11.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:49:11.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:49:11.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T04:49:11.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:49:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:49:11.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:49:11.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T04:49:11.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:49:11.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:49:11.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T04:49:11.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:49:11.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:49:11.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-02-05T04:49:13.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:49:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:49:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:49:13.564 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T04:49:13.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:13.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T04:49:13.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:49:13.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:49:13.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:49:13.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:49:13.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:13.593+0000 7fbd3d28c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:13.596 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:13.596+0000 7fbd3d28c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:13.598 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:13.597+0000 7fbd3d28c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:13.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:49:13.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:49:13.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:49:13.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:49:13.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:49:13.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:13.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:49:13.840 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:49:13.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:13.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:14.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:14.123 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:14.123+0000 7fbd3d28c4c0 -1 Falling back to public interface 2024-02-05T04:49:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:49:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:15.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:16.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:16.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:16.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:49:16.389 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:49:16.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:16.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:16.409 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:16.408+0000 7fbd3d28c4c0 -1 osd.1 31 log_to_monitors true 2024-02-05T04:49:16.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:17.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:17.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:17.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:49:17.699 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:49:17.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:17.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:17.972 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/506601281,v1:127.0.0.1:6811/506601281] [v2:127.0.0.1:6812/506601281,v1:127.0.0.1:6813/506601281] exists,up 13e5c017-87c5-4796-848d-b3d08e7401a3 2024-02-05T04:49:17.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:49:17.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:49:17.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:49:18.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:49:18.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:49:18.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:49:18.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:49:18.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:49:18.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:49:18.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:49:18.321 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:18.321 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:49:18.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:49:18.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:18.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:49:18.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-02-05T04:49:18.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-02-05T04:49:18.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-02-05T04:49:18.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:18.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:49:18.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-02-05T04:49:18.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-02-05T04:49:18.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-02-05T04:49:18.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:18.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:49:18.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T04:49:18.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T04:49:18.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509459' 2024-02-05T04:49:18.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:18.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-02-05T04:49:18.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:18.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:49:18.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-02-05T04:49:18.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:18.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-02-05T04:49:18.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-02-05T04:49:18.632 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 128849018885 2024-02-05T04:49:18.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:18.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-02-05T04:49:18.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:18.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-02-05T04:49:18.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:18.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:49:18.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-02-05T04:49:18.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:18.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-02-05T04:49:18.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-02-05T04:49:18.920 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 150323855362 2024-02-05T04:49:18.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:19.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-02-05T04:49:19.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:20.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:49:20.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:20.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-02-05T04:49:20.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:21.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:49:21.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:21.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-02-05T04:49:21.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:21.738 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T04:49:21.738 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:21.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:49:21.740 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T04:49:21.740 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:21.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T04:49:21.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T04:49:21.741 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509459 2024-02-05T04:49:21.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:49:22.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-02-05T04:49:22.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:49:22.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:22.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:22.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:49:22.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:49:22.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:49:22.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:49:22.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:49:22.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:49:22.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:49:22.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:49:22.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:49:22.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:49:22.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:22.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:23.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:49:23.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:49:23.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:49:23.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-02-05T04:49:23.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T04:49:23.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T04:49:23.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T04:49:23.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T04:49:23.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-02-05T04:49:23.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-02-05T04:49:23.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:49:23.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T04:49:23.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T04:49:23.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:49:23.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-02-05T04:49:23.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-02-05T04:49:23.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:23.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:23.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:23.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:23.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:23.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-02-05T04:49:23.900 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:49:23.900 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T04:49:23.901 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T04:49:23.901 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-22T04:47:43.900396+0000" 2024-02-05T04:49:23.901 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:49:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T04:49:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T04:49:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T04:49:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:49:23.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:49:23.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:23.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:23.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:23.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:24.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:48:37.048912+0000 '>' 2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:24.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:49:25.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:49:25.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:49:25.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:49:25.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:25.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:25.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:25.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:25.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:48:37.048912+0000 '>' 2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:25.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:49:26.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:49:26.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:49:26.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:49:26.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:26.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:26.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:26.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:26.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:48:37.048912+0000 '>' 2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:26.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:49:27.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:49:27.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:49:27.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:49:27.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:27.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:27.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:27.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:28.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:48:37.048912+0000 '>' 2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:28.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:49:29.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:49:29.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:49:29.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:49:29.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:29.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:29.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:29.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:49:25.696867+0000 '>' 2024-02-05T04:48:37.048912+0000 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:49:29.317 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:49:29.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:49:29.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:49:29.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:49:29.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:49:29.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:49:29.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:49:29.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:49:29.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:49:29.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:49:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:49:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:49:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:49:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:29.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:49:29.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018888 2024-02-05T04:49:29.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018888 2024-02-05T04:49:29.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888' 2024-02-05T04:49:29.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:29.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:49:29.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-02-05T04:49:29.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-02-05T04:49:29.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-02-05T04:49:29.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:29.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:49:29.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-02-05T04:49:29.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-02-05T04:49:29.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509462' 2024-02-05T04:49:29.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:29.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018888 2024-02-05T04:49:29.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:29.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:49:29.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018888 2024-02-05T04:49:29.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:29.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018888 2024-02-05T04:49:29.976 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 128849018888 2024-02-05T04:49:29.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-02-05T04:49:29.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:30.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018888 2024-02-05T04:49:30.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:31.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:49:31.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:31.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018888 2024-02-05T04:49:31.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:31.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-02-05T04:49:31.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:31.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:49:31.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-02-05T04:49:31.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:31.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-02-05T04:49:31.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-02-05T04:49:31.519 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 150323855365 2024-02-05T04:49:31.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:31.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-02-05T04:49:31.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:31.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-02-05T04:49:31.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:31.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:49:31.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-02-05T04:49:31.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:31.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-02-05T04:49:31.792 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509462 2024-02-05T04:49:31.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-02-05T04:49:31.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:49:32.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-02-05T04:49:32.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:49:32.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:32.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:49:32.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:49:32.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:49:32.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:49:32.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:32.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:33.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:49:33.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:49:33.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:49:33.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-02-05T04:49:33.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:49:33.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:49:33.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:49:33.274 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:33.274 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:49:33.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:49:33.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:33.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:49:33.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-02-05T04:49:33.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-02-05T04:49:33.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-02-05T04:49:33.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:33.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:49:33.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-02-05T04:49:33.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-02-05T04:49:33.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-02-05T04:49:33.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:33.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:49:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-02-05T04:49:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-02-05T04:49:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509464' 2024-02-05T04:49:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:33.580 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-02-05T04:49:33.580 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:33.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:49:33.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-02-05T04:49:33.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:33.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-02-05T04:49:33.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-02-05T04:49:33.583 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 128849018890 2024-02-05T04:49:33.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:33.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-02-05T04:49:33.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:34.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:49:34.849 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:35.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018890 2024-02-05T04:49:35.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:35.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855367 2024-02-05T04:49:35.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:35.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:49:35.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855367 2024-02-05T04:49:35.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:35.123 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 150323855367 2024-02-05T04:49:35.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-02-05T04:49:35.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-02-05T04:49:35.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:35.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-02-05T04:49:35.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:35.392 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-02-05T04:49:35.392 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:35.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:49:35.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-02-05T04:49:35.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:35.395 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509464 2024-02-05T04:49:35.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-02-05T04:49:35.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-02-05T04:49:35.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:49:35.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-02-05T04:49:35.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T04:49:35.665 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-02-05T04:49:35.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-02-05T04:49:35.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T04:49:35.920 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:49:35.920 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-02-05T04:49:25.696935+0000 21'10 36:94 [1,0] 1 [1,0] 1 21'10 2024-02-05T04:49:25.696867+0000 21'10 2024-02-05T04:49:25.696867+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:49:25.696867+0000 10 0 2024-02-05T04:49:35.920 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:49:35.920 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:49:35.921 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:49:35.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T04:49:35.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-02-05T04:49:36.180 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T04:49:36.192 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:49:36.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:49:36.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-02-05T04:49:36.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:49:36.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:49:36.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:49:36.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:49:36.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:49:37.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:49:37.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-02-05T04:49:37.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:49:37.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:49:37.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:49:37.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:49:37.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:49:37.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-02-05T04:49:38.206 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T04:49:38.206 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T04:49:38.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:38.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:49:38.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:49:38.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:49:38.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:49:38.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:49:38.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:49:38.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:49:38.477 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:49:38.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:38.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:49:38.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:49:38.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:49:38.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:49:38.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:49:38.506 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:38.505+0000 7f7813a0c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:38.508 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:38.507+0000 7f7813a0c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:38.509 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:38.508+0000 7f7813a0c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:39.023 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:39.023+0000 7f7813a0c4c0 -1 Falling back to public interface 2024-02-05T04:49:39.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:40.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:40.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:40.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:49:40.068 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:40.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:40.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:40.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:41.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:41.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:41.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:49:41.340 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:49:41.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:41.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:41.351 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:41.351+0000 7f7813a0c4c0 -1 osd.0 36 log_to_monitors true 2024-02-05T04:49:41.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:42.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:42.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:42.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:49:42.636 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:49:42.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:42.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:49:42.968 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/1377192985,v1:127.0.0.1:6803/1377192985] [v2:127.0.0.1:6804/1377192985,v1:127.0.0.1:6805/1377192985] exists,up ff8a9b9a-1dc9-4791-872d-d14529e2e327 2024-02-05T04:49:42.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:49:42.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:49:42.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:49:42.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:49:42.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:49:42.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:49:42.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:49:42.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:49:42.969 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:49:42.969 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:49:42.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:49:42.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:49:42.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:49:43.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:49:43.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:49:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:49:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:49:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:49:43.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:49:43.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:49:43.332 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:43.332 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:49:43.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:49:43.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:43.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:49:43.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-05T04:49:43.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-05T04:49:43.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-02-05T04:49:43.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:43.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:49:43.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-02-05T04:49:43.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-02-05T04:49:43.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-02-05T04:49:43.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:43.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:49:43.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-02-05T04:49:43.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-02-05T04:49:43.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509467' 2024-02-05T04:49:43.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:43.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-02-05T04:49:43.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:43.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:49:43.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-02-05T04:49:43.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:43.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-05T04:49:43.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-02-05T04:49:43.645 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 171798691842 2024-02-05T04:49:43.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:43.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-02-05T04:49:43.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:44.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:49:44.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:45.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-02-05T04:49:45.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:45.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-02-05T04:49:45.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:45.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:49:45.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-02-05T04:49:45.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-02-05T04:49:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-02-05T04:49:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 150323855370 2024-02-05T04:49:45.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:45.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-02-05T04:49:45.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:45.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-02-05T04:49:45.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:45.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:49:45.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-02-05T04:49:45.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:45.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-02-05T04:49:45.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-02-05T04:49:45.460 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509467 2024-02-05T04:49:45.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:49:45.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-02-05T04:49:45.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:49:45.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:45.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:49:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:49:46.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:49:46.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:49:46.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:49:46.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:49:46.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:49:46.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:49:46.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:49:46.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:49:46.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:46.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:46.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:49:46.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:49:46.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:49:46.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-02-05T04:49:46.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T04:49:46.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:49:46.680 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-02-05T04:49:46.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-02-05T04:49:46.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:49:46.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:49:46.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:49:46.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:49:47.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-02-05T04:49:47.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:49:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:49:47.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:49:47.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:49:47.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-02-05T04:49:48.430 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-02-05T04:49:48.430 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T04:49:48.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:48.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:49:48.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:49:48.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:49:48.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:49:48.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:49:48.949 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:49:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T04:49:48.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:49:48.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T04:49:48.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:49:48.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:49:48.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:49:48.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:49:48.980 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:48.978+0000 7ff5f37074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:48.982 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:48.981+0000 7ff5f37074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:48.983 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:48.982+0000 7ff5f37074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:49.504+0000 7ff5f37074c0 -1 Falling back to public interface 2024-02-05T04:49:49.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:50.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:50.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:50.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:49:50.524 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:50.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:50.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:50.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:49:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:49:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:51.819 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:49:51.819+0000 7ff5f37074c0 -1 osd.1 41 log_to_monitors true 2024-02-05T04:49:52.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:49:53.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:49:53.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:49:53.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:49:53.117 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:49:53.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:49:53.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/4135715298,v1:127.0.0.1:6811/4135715298] [v2:127.0.0.1:6812/4135715298,v1:127.0.0.1:6813/4135715298] exists,up 13e5c017-87c5-4796-848d-b3d08e7401a3 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:49:53.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:49:53.387 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:49:53.387 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:49:53.387 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:49:53.387 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:49:53.387 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:49:53.387 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:49:53.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:49:53.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:49:53.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:49:53.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:49:53.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:49:53.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:49:53.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:49:53.738 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:49:53.739 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:49:53.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:49:53.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:53.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:49:53.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-02-05T04:49:53.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-02-05T04:49:53.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-02-05T04:49:53.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:53.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:49:53.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-02-05T04:49:53.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-02-05T04:49:53.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-02-05T04:49:53.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:49:53.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:49:54.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509470 2024-02-05T04:49:54.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509470 2024-02-05T04:49:54.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509470' 2024-02-05T04:49:54.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:54.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-02-05T04:49:54.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:54.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:49:54.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-02-05T04:49:54.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:54.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-02-05T04:49:54.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-02-05T04:49:54.061 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 171798691845 2024-02-05T04:49:54.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:54.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-02-05T04:49:54.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:49:55.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:49:55.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:49:55.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-02-05T04:49:55.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:55.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-02-05T04:49:55.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:55.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:49:55.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-02-05T04:49:55.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-02-05T04:49:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-02-05T04:49:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 193273528322 2024-02-05T04:49:55.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:49:55.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-02-05T04:49:55.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:49:55.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509470 2024-02-05T04:49:55.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:49:55.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:49:55.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509470 2024-02-05T04:49:55.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:49:55.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509470 2024-02-05T04:49:55.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-02-05T04:49:55.878 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509470 2024-02-05T04:49:55.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:49:56.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509470 -lt 64424509470 2024-02-05T04:49:56.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:49:56.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:56.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:49:56.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:49:56.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:49:56.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:49:56.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:49:56.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:49:57.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:49:57.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:49:57.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:49:57.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-02-05T04:49:57.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T04:49:57.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T04:49:57.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:57.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:57.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:57.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:57.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T04:49:25.696867+0000 2024-02-05T04:49:57.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T04:49:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0 on osd.1 to repair 2024-02-05T04:49:57.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T04:49:25.696867+0000 2024-02-05T04:49:57.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T04:49:25.696867+0000 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:57.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:57.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:49:25.696867+0000 '>' 2024-02-05T04:49:25.696867+0000 2024-02-05T04:49:57.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:49:58.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:49:58.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:49:58.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:49:58.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:49:58.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:49:58.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:49:58.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:49:59.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:49:25.696867+0000 '>' 2024-02-05T04:49:25.696867+0000 2024-02-05T04:49:59.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:50:00.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:50:00.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:50:00.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:50:00.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:50:00.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:50:00.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:50:00.168 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:50:00.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:49:25.696867+0000 '>' 2024-02-05T04:49:25.696867+0000 2024-02-05T04:50:00.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:50:01.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:50:01.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:50:01.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:50:01.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:50:01.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:50:01.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:50:01.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:50:01.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:49:25.696867+0000 '>' 2024-02-05T04:49:25.696867+0000 2024-02-05T04:50:01.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:50:02.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:50:02.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:50:02.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:50:02.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:50:02.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:50:02.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:50:02.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:50:02.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:49:57.948793+0000 '>' 2024-02-05T04:49:25.696867+0000 2024-02-05T04:50:02.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T04:50:02.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-02-05T04:50:04.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-02-05T04:50:04.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:50:04.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:50:05.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:50:05.262 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:50:05.262 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:50:05.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:50:05.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:50:05.262 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:50:05.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691848 2024-02-05T04:50:05.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691848 2024-02-05T04:50:05.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848' 2024-02-05T04:50:05.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:50:05.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:50:05.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528325 2024-02-05T04:50:05.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528325 2024-02-05T04:50:05.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528325' 2024-02-05T04:50:05.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:50:05.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:50:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509473 2024-02-05T04:50:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509473 2024-02-05T04:50:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691848 1-193273528325 2-64424509473' 2024-02-05T04:50:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:50:05.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691848 2024-02-05T04:50:05.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:50:05.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:50:05.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691848 2024-02-05T04:50:05.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:50:05.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691848 2024-02-05T04:50:05.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691848' 2024-02-05T04:50:05.573 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 171798691848 2024-02-05T04:50:05.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:50:05.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691848 2024-02-05T04:50:05.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:50:06.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:50:06.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:50:07.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691848 -lt 171798691848 2024-02-05T04:50:07.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:50:07.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528325 2024-02-05T04:50:07.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:50:07.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:50:07.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528325 2024-02-05T04:50:07.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:50:07.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528325 2024-02-05T04:50:07.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-02-05T04:50:07.116 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 193273528325 2024-02-05T04:50:07.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:50:07.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528325 -lt 193273528325 2024-02-05T04:50:07.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:50:07.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509473 2024-02-05T04:50:07.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:50:07.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:50:07.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509473 2024-02-05T04:50:07.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:50:07.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509473 2024-02-05T04:50:07.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-02-05T04:50:07.386 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509473 2024-02-05T04:50:07.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:50:07.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509473 2024-02-05T04:50:07.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T04:50:07.906 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:50:07.906 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-02-05T04:49:57.948877+0000 21'10 46:131 [1,0] 1 [1,0] 1 21'10 2024-02-05T04:49:57.948793+0000 21'10 2024-02-05T04:49:57.948793+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:49:57.948793+0000 9 0 2024-02-05T04:50:07.906 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:50:07.906 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:50:07.907 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:50:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-02-05T04:50:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-02-05T04:50:08.167 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:50:08.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.512+0000 7fbd211ff640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:50:08.181 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:25.695+0000 7fbd211ff640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-02-05T04:50:08.182 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:57.947+0000 7ff5d767a640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-02-05T04:50:08.182 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:57.947+0000 7ff5d767a640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T04:50:08.182 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:57.947+0000 7ff5d767a640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T04:50:08.182 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:57.947+0000 7ff5d767a640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-02-05T04:50:08.182 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:49:57.947+0000 7ff5d767a640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:50:08.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:50:08.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:50:08.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:50:08.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:50:08.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:50:08.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:50:08.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:50:08.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:50:08.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:50:08.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:50:08.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:50:08.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:50:08.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:50:08.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:50:08.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:50:08.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:50:08.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:08.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:08.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:50:08.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:50:08.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:50:08.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:50:08.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:50:08.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:50:08.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:50:08.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:50:08.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:50:08.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:50:08.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:50:08.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:50:08.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:50:08.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:50:08.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:50:08.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:50:08.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:50:08.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:50:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:50:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:50:08.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:50:08.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:08.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:08.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:50:08.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:50:08.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:50:08.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:50:08.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:50:08.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:08.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:08.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-02-05T04:50:08.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-02-05T04:50:08.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T04:50:08.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:50:08.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:50:08.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:50:08.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T04:50:08.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:50:08.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T04:50:08.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:50:08.821 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:50:08.822 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:08.822 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:08.822 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:50:08.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:50:08.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T04:50:08.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:50:08.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:50:08.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:50:08.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:50:08.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:50:08.901 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:50:08.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T04:50:08.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-02-05T04:50:08.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T04:50:08.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:50:08.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:50:08.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:50:08.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T04:50:08.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:50:09.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:50:09.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:50:09.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:50:09.797 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:50:09.797 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:09.797 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:09.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:50:09.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:50:09.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:50:09.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:50:09.824 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-02-05T04:50:09.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-02-05T04:50:09.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:50:09.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:50:09.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:50:09.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:50:09.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:50:09.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:50:09.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:50:09.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:50:09.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:50:09.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:50:09.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51720807-d550-4862-926c-535430460d6f 2024-02-05T04:50:09.831 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 51720807-d550-4862-926c-535430460d6f 2024-02-05T04:50:09.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 51720807-d550-4862-926c-535430460d6f' 2024-02-05T04:50:09.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:50:09.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABacBl1sRqMhAATgTdy2kZAu0zjXkWQKcXbg== 2024-02-05T04:50:09.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABacBl1sRqMhAATgTdy2kZAu0zjXkWQKcXbg=="}' 2024-02-05T04:50:09.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51720807-d550-4862-926c-535430460d6f -i td/osd-scrub-repair/0/new.json 2024-02-05T04:50:10.052 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:50:10.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T04:50:10.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQABacBl1sRqMhAATgTdy2kZAu0zjXkWQKcXbg== --osd-uuid 51720807-d550-4862-926c-535430460d6f 2024-02-05T04:50:10.093 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:10.092+0000 7fe6d00144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:10.102 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:10.102+0000 7fe6d00144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:10.103 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:10.103+0000 7fe6d00144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:10.104 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:10.103+0000 7fe6d00144c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:50:10.104 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:10.103+0000 7fe6d00144c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:50:10.104 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:10.103+0000 7fe6d00144c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:50:10.195 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:10.194+0000 7fe6d00144c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:50:15.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T04:50:15.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:50:15.715 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:50:15.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:50:15.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:50:16.060 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:50:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:50:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:50:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:50:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:50:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:50:16.088 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:16.087+0000 7fb88c7744c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:16.090 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:16.090+0000 7fb88c7744c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:16.091 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:16.091+0000 7fb88c7744c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:16.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:50:16.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:50:16.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:50:16.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:50:16.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:50:16.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:16.328 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:50:16.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:50:16.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:16.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:16.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:16.612+0000 7fb88c7744c0 -1 Falling back to public interface 2024-02-05T04:50:17.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:17.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:17.597 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:50:17.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:50:17.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:17.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:17.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:18.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:18.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:18.873 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:50:18.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:50:18.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:18.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:18.981+0000 7fb88c7744c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:50:19.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:20.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:20.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:20.157 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:50:20.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:50:20.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:20.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:20.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:21.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:21.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:21.439 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:50:21.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:50:21.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:21.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:21.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:22.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:22.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:22.726 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:50:22.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:50:22.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:22.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:23.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:24.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:24.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:24.004 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:50:24.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:50:24.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:24.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:24.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:25.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:25.284 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:50:25.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:25.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:50:25.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:25.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:25.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:26.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:26.564 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:50:26.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:26.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:50:26.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:26.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:26.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:27.885 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:50:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:50:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:28.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:29.162 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:50:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:50:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:29.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:30.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:30.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:30.445 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:50:30.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:50:30.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:30.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:31.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:32.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:32.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:32.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:50:32.212 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:50:32.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:32.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:32.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:33.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:33.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:33.900 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:50:33.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:50:33.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:33.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:34.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:35.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:35.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:35.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:50:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:50:35.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:35.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:36.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:37.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:37.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:37.003 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:50:37.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:50:37.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:37.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:37.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:38.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:38.591 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:50:38.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:38.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:50:38.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:38.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:39.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:40.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:40.257 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:50:40.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:40.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:50:40.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:40.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:40.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:41.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:41.743 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:50:41.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:41.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:50:41.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:41.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:50:42.018 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3165947892,v1:127.0.0.1:6803/3165947892] [v2:127.0.0.1:6804/3165947892,v1:127.0.0.1:6805/3165947892] exists,up 51720807-d550-4862-926c-535430460d6f 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:50:42.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:50:42.024 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:50:42.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:50:42.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:50:42.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e817426b-0289-41aa-a49f-891d4faaa52f 2024-02-05T04:50:42.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e817426b-0289-41aa-a49f-891d4faaa52f' 2024-02-05T04:50:42.026 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 e817426b-0289-41aa-a49f-891d4faaa52f 2024-02-05T04:50:42.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:50:42.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAiacBlVBJVAhAAYXUyzCsPAdJNcvHwW/Zo/g== 2024-02-05T04:50:42.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAiacBlVBJVAhAAYXUyzCsPAdJNcvHwW/Zo/g=="}' 2024-02-05T04:50:42.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e817426b-0289-41aa-a49f-891d4faaa52f -i td/osd-scrub-repair/1/new.json 2024-02-05T04:50:42.352 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:50:42.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T04:50:42.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAiacBlVBJVAhAAYXUyzCsPAdJNcvHwW/Zo/g== --osd-uuid e817426b-0289-41aa-a49f-891d4faaa52f 2024-02-05T04:50:42.394 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:42.393+0000 7ff4918144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:42.437 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:42.436+0000 7ff4918144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:42.437 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:42.437+0000 7ff4918144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:42.438 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:42.437+0000 7ff4918144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:50:42.438 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:42.437+0000 7ff4918144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:50:42.438 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:42.437+0000 7ff4918144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:50:42.503 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:42.502+0000 7ff4918144c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:50:48.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T04:50:48.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:50:48.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:50:48.220 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:50:48.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:50:48.567 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:50:48.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:50:48.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:50:48.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:50:48.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:50:48.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:50:48.595 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:48.594+0000 7f84db40c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:48.597 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:48.596+0000 7f84db40c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:48.597+0000 7f84db40c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:50:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:50:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:50:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:50:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:50:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:50:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:50:48.835 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:50:48.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:48.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:49.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:49.117 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:49.116+0000 7f84db40c4c0 -1 Falling back to public interface 2024-02-05T04:50:50.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:50.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:50.103 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:50:50.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:50:50.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:50.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:50.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:51.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:51.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:51.379 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:50:51.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:50:51.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:51.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:51.445 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:50:51.443+0000 7f84db40c4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:50:51.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:52.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:52.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:52.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:50:52.658 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:50:52.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:52.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:52.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:53.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:53.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:53.935 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:50:53.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:50:53.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:53.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:54.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:55.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:55.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:55.250 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:50:55.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:50:55.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:55.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:55.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:56.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:56.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:56.529 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:50:56.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:50:56.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:56.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:56.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:57.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:57.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:57.808 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:50:57.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:50:57.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:57.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:58.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:50:59.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:50:59.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:50:59.202 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:50:59.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:50:59.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:50:59.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:50:59.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:51:00.482 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:51:00.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:00.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:00.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:01.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:01.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:01.775 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:51:01.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:51:01.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:01.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:03.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:03.097 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:51:03.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:03.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:51:03.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:03.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:03.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:04.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:04.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:04.753 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:51:04.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:51:04.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:04.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:05.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:06.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:06.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:06.031 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:51:06.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:51:06.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:06.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:06.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:07.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:07.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:07.926 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:51:07.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:51:07.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:07.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:09.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:09.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:09.367 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:51:09.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:51:09.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:09.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:10.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:11.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:11.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:11.066 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:51:11.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:51:11.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:11.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:11.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:12.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:12.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:12.359 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:51:12.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:51:12.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:12.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:13.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:14.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:14.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:14.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:51:14.349 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:51:14.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:14.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:51:14.621 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1826574701,v1:127.0.0.1:6811/1826574701] [v2:127.0.0.1:6812/1826574701,v1:127.0.0.1:6813/1826574701] exists,up e817426b-0289-41aa-a49f-891d4faaa52f 2024-02-05T04:51:14.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:51:14.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:51:14.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:51:14.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-02-05T04:51:14.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:51:14.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:51:14.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:51:14.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:51:14.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:51:14.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:51:14.627 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:51:14.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:51:14.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:51:14.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5fc3819d-8b88-4fea-825e-d29233e7194f 2024-02-05T04:51:14.630 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 5fc3819d-8b88-4fea-825e-d29233e7194f 2024-02-05T04:51:14.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5fc3819d-8b88-4fea-825e-d29233e7194f' 2024-02-05T04:51:14.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:51:14.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCacBlOV83JhAAqJfp5jGoLvkXxS6QfeZPzw== 2024-02-05T04:51:14.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCacBlOV83JhAAqJfp5jGoLvkXxS6QfeZPzw=="}' 2024-02-05T04:51:14.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5fc3819d-8b88-4fea-825e-d29233e7194f -i td/osd-scrub-repair/2/new.json 2024-02-05T04:51:14.981 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:51:14.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T04:51:14.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBCacBlOV83JhAAqJfp5jGoLvkXxS6QfeZPzw== --osd-uuid 5fc3819d-8b88-4fea-825e-d29233e7194f 2024-02-05T04:51:15.022 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:15.021+0000 7fe2b258b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:51:15.066 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:15.065+0000 7fe2b258b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:51:15.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:15.066+0000 7fe2b258b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:51:15.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:15.066+0000 7fe2b258b4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:51:15.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:15.066+0000 7fe2b258b4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:51:15.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:15.066+0000 7fe2b258b4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:51:15.141 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:15.139+0000 7fe2b258b4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T04:51:21.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T04:51:21.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:51:21.209 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:51:21.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:51:21.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:51:21.552 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:51:21.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:51:21.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:51:21.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:51:21.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:51:21.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:51:21.580 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:21.579+0000 7fe4a5a0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:51:21.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:21.581+0000 7fe4a5a0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:51:21.583 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:21.582+0000 7fe4a5a0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:51:21.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:51:21.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:51:21.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:51:21.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:51:21.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:51:21.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:21.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:51:21.851 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:51:21.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:21.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:22.102 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:22.101+0000 7fe4a5a0e4c0 -1 Falling back to public interface 2024-02-05T04:51:22.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:23.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:23.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:23.121 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:51:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:51:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:23.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:24.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:24.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:24.391 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:51:24.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:51:24.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:24.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:24.399 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:51:24.398+0000 7fe4a5a0e4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:51:24.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:25.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:25.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:25.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:51:25.709 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:51:25.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:25.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:26.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:27.031 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:51:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:51:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:27.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:28.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:28.385 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:51:28.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:28.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:51:28.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:28.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:28.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:29.666 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:51:29.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:51:29.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:29.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:29.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:30.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:30.945 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:51:30.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:30.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:51:30.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:30.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:31.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:32.225 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:51:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:51:32.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:32.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:32.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:33.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:33.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:33.504 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:51:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:51:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:33.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:34.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:34.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:34.781 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:51:34.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:51:34.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:34.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:35.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:36.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:36.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:36.061 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:51:36.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:51:36.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:36.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:36.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:37.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:37.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:37.339 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:51:37.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:51:37.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:37.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:37.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:38.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:38.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:38.992 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:51:38.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:51:38.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:38.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:39.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:40.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:40.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:40.723 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:51:40.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:51:40.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:40.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:41.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:42.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:42.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:42.355 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:51:42.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:51:42.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:42.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:42.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:43.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:43.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:43.955 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:51:43.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:51:43.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:43.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:44.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:45.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:45.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:45.415 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:51:45.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:51:45.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:45.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:46.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:51:47.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:51:47.234 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:51:47.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:51:47.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:51:47.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:51:47.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:51:47.613 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3524015365,v1:127.0.0.1:6819/3524015365] [v2:127.0.0.1:6820/3524015365,v1:127.0.0.1:6821/3524015365] exists,up 5fc3819d-8b88-4fea-825e-d29233e7194f 2024-02-05T04:51:47.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:51:47.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:51:47.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:51:47.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-02-05T04:51:47.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T04:51:48.028 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'testpool' created 2024-02-05T04:51:48.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:51:49.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-02-05T04:51:49.422 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 2 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:51:49.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:51:49.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:51:49.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:51:49.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:51:49.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:51:49.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:51:49.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:51:49.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:51:49.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:51:49.875 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:51:49.875 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:51:49.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:51:49.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:51:49.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:51:50.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-02-05T04:51:50.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-02-05T04:51:50.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-02-05T04:51:50.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:51:50.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:51:50.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T04:51:50.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T04:51:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972' 2024-02-05T04:51:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:51:50.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:51:50.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-02-05T04:51:50.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-02-05T04:51:50.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972 2-64424509445' 2024-02-05T04:51:50.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:51:50.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-02-05T04:51:50.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:51:50.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:51:50.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-02-05T04:51:50.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:51:50.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-02-05T04:51:50.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-02-05T04:51:50.330 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836501 2024-02-05T04:51:50.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:51:50.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-02-05T04:51:50.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:51:51.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:51:51.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:51:51.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836501 2024-02-05T04:51:51.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:51:51.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T04:51:51.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:51:51.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:51:51.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T04:51:51.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:51:51.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T04:51:51.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T04:51:51.897 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672972 2024-02-05T04:51:51.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:51:52.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T04:51:52.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:51:52.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-02-05T04:51:52.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:51:52.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:51:52.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-02-05T04:51:52.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:51:52.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-02-05T04:51:52.165 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509445 2024-02-05T04:51:52.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-02-05T04:51:52.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:51:52.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-02-05T04:51:52.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:51:52.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:51:52.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:51:52.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:51:52.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:51:52.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:51:52.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:51:52.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:51:52.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:51:52.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:51:52.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:51:53.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:51:53.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:51:53.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:51:53.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:51:53.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:51:53.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T04:51:53.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T04:51:53.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T04:51:53.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:51:53.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T04:51:53.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:51:53.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:51:53.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:51:53.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:51:53.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:51:53.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:51:53.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:51:53.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:51:53.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:51:53.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:51:53.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:51:53.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T04:51:54.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T04:51:54.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T04:51:54.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T04:51:54.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T04:51:54.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T04:51:54.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:51:54.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T04:51:54.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:51:54.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:51:54.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:51:54.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:51:54.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:51:54.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:51:54.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:51:54.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:51:54.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:51:54.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:51:54.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:51:54.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:51:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:51:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T04:51:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T04:51:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T04:51:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T04:51:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T04:51:55.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T04:51:55.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T04:51:55.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T04:51:55.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T04:51:55.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T04:51:55.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T04:51:55.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:51:55.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:51:55.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:51:56.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:51:56.222 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:51:56.222 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:51:56.222 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:51:56.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:51:56.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:51:56.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:51:56.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-02-05T04:51:56.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-02-05T04:51:56.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-02-05T04:51:56.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:56.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:56.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:56.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:56.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:56.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:56.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:56.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:57.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:57.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:57.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:57.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:57.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:57.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:57.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:57.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:57.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:57.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:57.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-02-05T04:51:57.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:51:57.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:51:58.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:51:58.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:51:58.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:51:58.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:51:58.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:51:58.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:51:58.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:51:58.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:51:58.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:51:58.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:51:58.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:51:58.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:51:58.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-02-05T04:51:58.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:51:58.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:51:58.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:51:58.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:51:58.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:51:58.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-02-05T04:52:00.098 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T04:52:00.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:52:00.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:52:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:52:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:52:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:52:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:52:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:52:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:52:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:52:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:00.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:52:00.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:52:00.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:52:00.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:52:00.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:52:00.653 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:00.652+0000 7f5e626164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:00.655 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:00.654+0000 7f5e626164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:00.656 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:00.655+0000 7f5e626164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:52:00.902 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:52:00.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:00.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:01.174 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:01.173+0000 7f5e626164c0 -1 Falling back to public interface 2024-02-05T04:52:01.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:02.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:02.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:02.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:52:02.190 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:02.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:02.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:02.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:03.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:03.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:03.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:52:03.461 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:52:03.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:03.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:03.512 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:03.510+0000 7f5e626164c0 -1 osd.0 20 log_to_monitors true 2024-02-05T04:52:03.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:04.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:04.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:04.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:52:04.738 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:52:04.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:04.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:05.013 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/1387003311,v1:127.0.0.1:6803/1387003311] [v2:127.0.0.1:6804/1387003311,v1:127.0.0.1:6805/1387003311] exists,up 51720807-d550-4862-926c-535430460d6f 2024-02-05T04:52:05.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:52:05.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:52:05.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:52:05.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:52:05.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:52:05.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:52:05.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:52:05.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:52:05.014 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:52:05.014 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:52:05.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:52:05.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:52:05.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:52:05.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:52:05.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:52:05.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:52:05.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:52:05.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:52:05.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:52:05.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:52:05.363 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:05.363 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:52:05.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:52:05.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:05.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:52:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-02-05T04:52:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-02-05T04:52:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-02-05T04:52:05.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:05.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:52:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T04:52:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T04:52:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672976' 2024-02-05T04:52:05.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:05.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:52:05.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T04:52:05.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T04:52:05.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672976 2-64424509449' 2024-02-05T04:52:05.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:05.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-02-05T04:52:05.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:05.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:52:05.673 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-02-05T04:52:05.673 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:05.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-02-05T04:52:05.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-02-05T04:52:05.674 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 103079215106 2024-02-05T04:52:05.674 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:05.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-02-05T04:52:05.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:05.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T04:52:05.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:05.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:52:05.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T04:52:05.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:05.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T04:52:05.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T04:52:05.944 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672976 2024-02-05T04:52:05.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:52:06.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T04:52:06.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:06.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T04:52:06.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:06.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:52:06.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T04:52:06.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:06.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T04:52:06.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T04:52:06.215 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509449 2024-02-05T04:52:06.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:52:06.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T04:52:06.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:52:06.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:06.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:06.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:52:06.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:06.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:06.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:06.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:06.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:06.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:06.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:07.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:52:07.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:07.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:07.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:07.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:52:07.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:52:07.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:52:07.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-02-05T04:52:07.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:52:07.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:52:07.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:52:07.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:52:07.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T04:52:07.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:52:07.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:52:07.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:52:07.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:52:07.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:52:08.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-02-05T04:52:09.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:09.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:52:09.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:52:09.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:52:09.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:52:09.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:52:09.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:52:09.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:52:09.679 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T04:52:09.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:09.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T04:52:09.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:52:09.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:52:09.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:52:09.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:52:09.708 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:09.706+0000 7f79c300e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:09.710 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:09.709+0000 7f79c300e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:09.712 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:09.710+0000 7f79c300e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:52:10.228 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:10.227+0000 7f79c300e4c0 -1 Falling back to public interface 2024-02-05T04:52:10.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:11.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:52:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:11.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:52:11.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:12.532 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:12.531+0000 7f79c300e4c0 -1 osd.1 25 log_to_monitors true 2024-02-05T04:52:12.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:12.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:12.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:52:12.541 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:52:12.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:12.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:52:12.913 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 29 up_thru 24 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/500382763,v1:127.0.0.1:6811/500382763] [v2:127.0.0.1:6812/500382763,v1:127.0.0.1:6813/500382763] exists,up e817426b-0289-41aa-a49f-891d4faaa52f 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:52:12.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:52:12.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:52:12.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:52:12.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:52:12.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:52:12.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:52:12.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:52:13.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:52:13.274 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:13.274 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:52:13.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:52:13.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:13.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:52:13.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215108 2024-02-05T04:52:13.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215108 2024-02-05T04:52:13.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108' 2024-02-05T04:52:13.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:13.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:52:13.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T04:52:13.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T04:52:13.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-124554051586' 2024-02-05T04:52:13.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:13.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:52:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T04:52:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T04:52:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-124554051586 2-64424509452' 2024-02-05T04:52:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:13.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215108 2024-02-05T04:52:13.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:13.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:52:13.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215108 2024-02-05T04:52:13.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:13.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215108 2024-02-05T04:52:13.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-02-05T04:52:13.607 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 103079215108 2024-02-05T04:52:13.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:13.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215108 2024-02-05T04:52:13.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:13.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-02-05T04:52:13.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:13.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:52:13.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-02-05T04:52:13.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:13.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T04:52:13.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-02-05T04:52:13.881 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 124554051586 2024-02-05T04:52:13.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:52:14.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T04:52:14.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:14.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T04:52:14.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:14.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:52:14.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T04:52:14.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:14.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T04:52:14.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T04:52:14.186 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509452 2024-02-05T04:52:14.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:52:14.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T04:52:14.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:52:14.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:14.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:52:15.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:15.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:15.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:52:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T04:52:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T04:52:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T04:52:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:52:15.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:15.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:52:15.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:15.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:15.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T04:52:16.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T04:52:16.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T04:52:16.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T04:52:16.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T04:52:16.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:52:16.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:16.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T04:52:16.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:16.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:16.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T04:52:17.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T04:52:17.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T04:52:17.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T04:52:17.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T04:52:17.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T04:52:17.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T04:52:17.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T04:52:17.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:17.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:17.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:17.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:17.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:17.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:17.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:18.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:52:18.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:18.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:18.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:18.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:52:18.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:52:18.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:52:18.553 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:52:18.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:52:18.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:52:18.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:52:19.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-02-05T04:52:19.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:52:19.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:52:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T04:52:20.699 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T04:52:21.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:21.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:52:21.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:52:21.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:52:21.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:52:21.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:52:21.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:52:21.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:52:21.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:52:21.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:52:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:52:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:52:21.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:21.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:52:21.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:52:21.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:52:21.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:52:21.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:52:21.248 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:21.247+0000 7f61c14a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:21.250 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:21.249+0000 7f61c14a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:21.251 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:21.250+0000 7f61c14a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:21.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:52:21.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:52:21.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:52:21.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:52:21.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:52:21.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:21.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:52:21.496 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:52:21.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:21.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:21.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:21.773 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:21.772+0000 7f61c14a14c0 -1 Falling back to public interface 2024-02-05T04:52:22.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:22.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:22.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:52:22.772 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:22.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:22.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:23.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:24.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:24.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:24.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:52:24.051 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:52:24.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:24.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:24.197 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:24.196+0000 7f61c14a14c0 -1 osd.0 30 log_to_monitors true 2024-02-05T04:52:24.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:25.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:25.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:25.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:52:25.352 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:52:25.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:25.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:52:25.620 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 34 up_thru 26 down_at 31 last_clean_interval [24,30) [v2:127.0.0.1:6802/483110846,v1:127.0.0.1:6803/483110846] [v2:127.0.0.1:6804/483110846,v1:127.0.0.1:6805/483110846] exists,up 51720807-d550-4862-926c-535430460d6f 2024-02-05T04:52:25.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:52:25.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:52:25.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:52:25.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:52:25.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:52:25.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:52:25.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:52:25.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:52:25.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:52:25.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:52:25.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:52:25.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:52:25.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:52:25.984 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:25.984 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:52:25.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:52:25.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:25.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:52:26.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T04:52:26.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T04:52:26.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-02-05T04:52:26.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:26.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:52:26.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-02-05T04:52:26.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-02-05T04:52:26.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589' 2024-02-05T04:52:26.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:26.195 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:52:26.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-05T04:52:26.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-05T04:52:26.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589 2-64424509455' 2024-02-05T04:52:26.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:26.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-02-05T04:52:26.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:26.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:52:26.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-02-05T04:52:26.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:26.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T04:52:26.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-02-05T04:52:26.301 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 146028888066 2024-02-05T04:52:26.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:26.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-02-05T04:52:26.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:52:27.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:52:27.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:27.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-02-05T04:52:27.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:52:28.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:52:28.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:29.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-02-05T04:52:29.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:29.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051589 2024-02-05T04:52:29.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:29.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:52:29.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051589 2024-02-05T04:52:29.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:29.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-02-05T04:52:29.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051589' 2024-02-05T04:52:29.132 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 124554051589 2024-02-05T04:52:29.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:52:29.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051589 2024-02-05T04:52:29.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:29.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-05T04:52:29.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:29.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:52:29.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-05T04:52:29.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-05T04:52:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-05T04:52:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509455 2024-02-05T04:52:29.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:52:29.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-05T04:52:29.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:52:29.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:29.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:30.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:52:30.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:30.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:30.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:30.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:30.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:30.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:30.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:30.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:52:30.278 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:30.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:30.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:30.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:52:30.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:52:30.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:52:30.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-02-05T04:52:30.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:52:30.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:52:30.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:52:30.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:52:30.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:52:30.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:52:30.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:52:30.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:52:30.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:52:31.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:52:31.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-02-05T04:52:31.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:52:31.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:52:31.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T04:52:31.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:52:31.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:52:31.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-02-05T04:52:32.454 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-02-05T04:52:32.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:32.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:52:32.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:52:32.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:52:32.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:52:33.004 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:33.002+0000 7f94c56154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:33.006 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:33.005+0000 7f94c56154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:33.007 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:33.006+0000 7f94c56154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:33.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:33.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:52:33.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:33.538 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:33.537+0000 7f94c56154c0 -1 Falling back to public interface 2024-02-05T04:52:34.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:34.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:34.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:52:34.518 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:34.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:34.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:52:34.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:35.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:35.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:35.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:52:35.796 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:52:35.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:35.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:52:35.852 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:35.850+0000 7f94c56154c0 -1 osd.1 35 log_to_monitors true 2024-02-05T04:52:36.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:52:37.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:52:37.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:52:37.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:52:37.126 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:52:37.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:52:37.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:52:37.402 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6810/693587929,v1:127.0.0.1:6811/693587929] [v2:127.0.0.1:6812/693587929,v1:127.0.0.1:6813/693587929] exists,up e817426b-0289-41aa-a49f-891d4faaa52f 2024-02-05T04:52:37.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:52:37.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:52:37.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:52:37.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:52:37.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:52:37.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:52:37.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:52:37.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:52:37.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:52:37.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:52:37.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:52:37.753 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:37.753 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:52:37.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:52:37.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:37.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:52:37.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-02-05T04:52:37.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-02-05T04:52:37.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069' 2024-02-05T04:52:37.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:37.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:52:37.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-02-05T04:52:37.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-02-05T04:52:37.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546' 2024-02-05T04:52:37.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:37.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:52:38.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T04:52:38.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T04:52:38.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546 2-64424509459' 2024-02-05T04:52:38.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:38.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888069 2024-02-05T04:52:38.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:38.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:52:38.066 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888069 2024-02-05T04:52:38.066 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:38.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-02-05T04:52:38.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2024-02-05T04:52:38.067 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 146028888069 2024-02-05T04:52:38.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:38.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888069 2024-02-05T04:52:38.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:52:39.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:52:39.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:39.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888069 2024-02-05T04:52:39.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:52:40.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:52:40.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:40.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888070 -lt 146028888069 2024-02-05T04:52:40.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:40.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-02-05T04:52:40.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:40.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:52:40.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-02-05T04:52:40.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:40.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-02-05T04:52:40.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-02-05T04:52:40.884 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 167503724546 2024-02-05T04:52:40.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:52:41.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-02-05T04:52:41.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:41.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T04:52:41.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:41.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:52:41.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T04:52:41.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:41.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T04:52:41.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T04:52:41.161 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509459 2024-02-05T04:52:41.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:52:41.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-02-05T04:52:41.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:52:41.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:41.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:41.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:52:41.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:41.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:41.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:41.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:41.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:41.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:41.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:42.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:52:42.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:42.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:42.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:42.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:52:42.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:52:42.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:52:42.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-02-05T04:52:42.450 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T04:52:42.450 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T04:52:42.450 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:52:42.464 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T04:52:42.464 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T04:52:42.464 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:52:42.478 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2: { 2024-02-05T04:52:42.478 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-02-05T04:52:42.478 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:52:42.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-02-05T04:52:42.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T04:52:42.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T04:52:42.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T04:52:42.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T04:52:42.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-02-05T04:52:42.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-02-05T04:52:42.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:52:42.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T04:52:42.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T04:52:42.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:52:43.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-02-05T04:52:43.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-02-05T04:52:43.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:52:43.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:52:43.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:52:43.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:52:43.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:43.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-02-05T04:52:43.448 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:52:43.448 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T04:52:43.448 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T04:52:43.448 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-22T04:51:03.447788+0000" 2024-02-05T04:52:43.448 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:52:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T04:52:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T04:52:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T04:52:43.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:52:43.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:52:43.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:52:43.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:52:43.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:52:43.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:52:43.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:51:47.987699+0000 '>' 2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:43.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:52:44.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:52:44.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:52:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:52:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:52:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:52:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:52:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:52:44.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:51:47.987699+0000 '>' 2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:44.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:52:45.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:52:45.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:52:45.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:52:45.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:52:45.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:52:45.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:52:45.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:52:46.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:51:47.987699+0000 '>' 2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:46.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:52:47.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:52:47.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:52:47.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:52:47.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:52:47.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:52:47.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:52:47.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:52:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:51:47.987699+0000 '>' 2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:52:48.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:52:48.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:52:48.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:52:48.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:52:48.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:52:48.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:52:48.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:52:48.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:52:43.937769+0000 '>' 2024-02-05T04:51:47.987699+0000 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:52:48.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:52:48.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:52:48.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:52:48.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:52:48.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:52:48.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:52:48.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:52:49.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:52:49.178 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:49.179 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:52:49.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:52:49.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:49.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:52:49.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888072 2024-02-05T04:52:49.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888072 2024-02-05T04:52:49.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888072' 2024-02-05T04:52:49.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:49.284 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:52:49.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724549 2024-02-05T04:52:49.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724549 2024-02-05T04:52:49.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888072 1-167503724549' 2024-02-05T04:52:49.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:49.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:52:49.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-02-05T04:52:49.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-02-05T04:52:49.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888072 1-167503724549 2-64424509462' 2024-02-05T04:52:49.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:49.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888072 2024-02-05T04:52:49.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:49.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:52:49.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888072 2024-02-05T04:52:49.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:49.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888072 2024-02-05T04:52:49.492 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 146028888072 2024-02-05T04:52:49.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888072' 2024-02-05T04:52:49.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:49.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888071 -lt 146028888072 2024-02-05T04:52:49.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:52:50.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:52:50.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:51.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888073 -lt 146028888072 2024-02-05T04:52:51.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:51.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724549 2024-02-05T04:52:51.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:51.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:52:51.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724549 2024-02-05T04:52:51.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:51.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724549 2024-02-05T04:52:51.037 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 167503724549 2024-02-05T04:52:51.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-02-05T04:52:51.037 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:52:51.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724549 -lt 167503724549 2024-02-05T04:52:51.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:51.307 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-02-05T04:52:51.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:51.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:52:51.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-02-05T04:52:51.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:51.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-02-05T04:52:51.310 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509462 2024-02-05T04:52:51.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-02-05T04:52:51.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:52:51.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-02-05T04:52:51.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:52:51.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:51.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:51.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:52:51.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:52:51.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:52:51.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:52:51.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:52:51.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:52:51.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:52:51.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:52:52.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:52:52.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:52:52.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:52:52.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:52:52.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:52:52.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:52:52.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:52:52.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-02-05T04:52:52.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:52:52.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:52:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:52:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:52:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:52:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:52:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:52.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:52:52.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888074 2024-02-05T04:52:52.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888074 2024-02-05T04:52:52.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074' 2024-02-05T04:52:52.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:52.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:52:53.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724551 2024-02-05T04:52:53.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724551 2024-02-05T04:52:53.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074 1-167503724551' 2024-02-05T04:52:53.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:52:53.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:52:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-02-05T04:52:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-02-05T04:52:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074 1-167503724551 2-64424509464' 2024-02-05T04:52:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:53.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888074 2024-02-05T04:52:53.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:53.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:52:53.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888074 2024-02-05T04:52:53.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:53.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888074 2024-02-05T04:52:53.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888074' 2024-02-05T04:52:53.139 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 146028888074 2024-02-05T04:52:53.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:53.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888073 -lt 146028888074 2024-02-05T04:52:53.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:52:54.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:52:54.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:52:54.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888074 -lt 146028888074 2024-02-05T04:52:54.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:54.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724551 2024-02-05T04:52:54.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:54.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:52:54.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724551 2024-02-05T04:52:54.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:54.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724551 2024-02-05T04:52:54.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2024-02-05T04:52:54.687 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 167503724551 2024-02-05T04:52:54.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:52:54.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724551 -lt 167503724551 2024-02-05T04:52:54.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:52:54.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-02-05T04:52:54.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:52:54.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:52:54.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-02-05T04:52:54.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:52:54.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-02-05T04:52:54.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-02-05T04:52:54.956 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509464 2024-02-05T04:52:54.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:52:55.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-02-05T04:52:55.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-02-05T04:52:55.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-02-05T04:52:55.477 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:52:55.477 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-02-05T04:52:43.937852+0000 20'10 40:96 [1,0] 1 [1,0] 1 20'10 2024-02-05T04:52:43.937769+0000 20'10 2024-02-05T04:52:43.937769+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:52:43.937769+0000 10 0 2024-02-05T04:52:55.478 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:52:55.478 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:52:55.478 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:52:55.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-02-05T04:52:55.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-02-05T04:52:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:52:55.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:52:55.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:52:55.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:52:55.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:52:55.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:52:55.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:52:55.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:52:55.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:52:55.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:52:55.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:52:55.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:52:55.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:52:55.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:52:55.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:52:55.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:52:55.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:52:55.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:52:55.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:55.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:55.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:52:55.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:52:55.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:52:55.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:52:55.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:52:55.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:52:55.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:52:55.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:52:55.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:52:55.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:52:55.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:52:55.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:52:55.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:52:55.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:52:55.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:52:55.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:52:55.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:52:55.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:52:55.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:52:55.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:52:55.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:52:55.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:55.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:55.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:52:55.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:52:55.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:52:55.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:52:55.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:52:55.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:55.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:55.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:52:55.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:52:55.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T04:52:55.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T04:52:56.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:52:56.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:52:56.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:52:56.269 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:52:56.269 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:56.269 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:56.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:56.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T04:52:56.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:52:56.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:52:56.464 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:52:56.464 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:56.464 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:56.464 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:52:56.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:52:56.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T04:52:56.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:52:56.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:52:56.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:52:56.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:52:56.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T04:52:56.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-02-05T04:52:56.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T04:52:56.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:52:56.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:52:56.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:52:56.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T04:52:56.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:52:57.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:52:57.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:52:57.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:52:57.430 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:52:57.430 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:57.431 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:57.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:57.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:52:57.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:52:57.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T04:52:57.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:52:57.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:52:57.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:52:57.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:52:57.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584 2024-02-05T04:52:57.465 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584 2024-02-05T04:52:57.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584' 2024-02-05T04:52:57.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:52:57.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpacBlFj+NHBAA4OSe8fPsRc4IhoIDtZWAKw== 2024-02-05T04:52:57.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpacBlFj+NHBAA4OSe8fPsRc4IhoIDtZWAKw=="}' 2024-02-05T04:52:57.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584 -i td/osd-scrub-repair/0/new.json 2024-02-05T04:52:57.669 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:52:57.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T04:52:57.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQCpacBlFj+NHBAA4OSe8fPsRc4IhoIDtZWAKw== --osd-uuid 7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584 2024-02-05T04:52:57.711 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:57.709+0000 7fa026a634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:57.712 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:57.711+0000 7fa026a634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:57.713 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:57.712+0000 7fa026a634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:52:57.714 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:57.713+0000 7fa026a634c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:52:57.714 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:57.713+0000 7fa026a634c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:52:57.714 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:57.713+0000 7fa026a634c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:52:57.813 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:52:57.811+0000 7fa026a634c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:53:03.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T04:53:03.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:53:03.189 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:53:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:53:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:53:03.566 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:53:03.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:53:03.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:53:03.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:53:03.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:53:03.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:53:03.595 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:03.594+0000 7f1f096144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:03.597 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:03.596+0000 7f1f096144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:03.598 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:03.597+0000 7f1f096144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:03.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:53:03.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:53:03.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:53:03.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:53:03.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:53:03.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:03.873 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:53:03.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:53:03.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:03.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:04.116 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:04.115+0000 7f1f096144c0 -1 Falling back to public interface 2024-02-05T04:53:04.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:05.140 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:53:05.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:53:05.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:05.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:05.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:06.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:06.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:06.411 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:53:06.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:53:06.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:06.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:06.464 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:06.462+0000 7f1f096144c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:53:06.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:07.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:07.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:07.716 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:53:07.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:53:07.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:07.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:07.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:08.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:08.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:08.998 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:53:08.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:53:08.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:08.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:10.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:10.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:10.278 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:53:10.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:53:10.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:10.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:10.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:11.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:11.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:11.557 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:53:11.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:53:11.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:11.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:11.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:12.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:12.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:12.844 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:53:12.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:53:12.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:12.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:13.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:14.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:14.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:14.121 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:53:14.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:53:14.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:14.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:14.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:15.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:15.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:15.404 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:53:15.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:53:15.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:15.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:15.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:16.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:16.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:16.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:53:16.679 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:53:16.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:16.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:16.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:17.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:17.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:17.957 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:53:17.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:53:17.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:17.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:18.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:19.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:19.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:19.654 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:53:19.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:53:19.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:19.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:20.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:21.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:21.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:21.131 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:53:21.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:53:21.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:21.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:21.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:22.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:22.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:22.875 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:53:22.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:53:22.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:22.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:23.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:24.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:24.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:24.353 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:53:24.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:53:24.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:24.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:24.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:25.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:25.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:25.778 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:53:25.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:53:25.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:25.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:26.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:27.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:27.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:27.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:53:27.404 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:53:27.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:27.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:28.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:29.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:29.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:29.035 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:53:29.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:53:29.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:29.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:53:29.308 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1858626428,v1:127.0.0.1:6803/1858626428] [v2:127.0.0.1:6804/1858626428,v1:127.0.0.1:6805/1858626428] exists,up 7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:53:29.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:53:29.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T04:53:29.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:53:29.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:53:29.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=07250185-8ea5-405f-a4ad-49b8b32635cc 2024-02-05T04:53:29.315 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 07250185-8ea5-405f-a4ad-49b8b32635cc 2024-02-05T04:53:29.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 07250185-8ea5-405f-a4ad-49b8b32635cc' 2024-02-05T04:53:29.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:53:29.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJacBlcHWLExAA5lHyQep7seO+aZT2ADFbLg== 2024-02-05T04:53:29.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJacBlcHWLExAA5lHyQep7seO+aZT2ADFbLg=="}' 2024-02-05T04:53:29.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 07250185-8ea5-405f-a4ad-49b8b32635cc -i td/osd-scrub-repair/1/new.json 2024-02-05T04:53:29.711 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:53:29.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T04:53:29.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDJacBlcHWLExAA5lHyQep7seO+aZT2ADFbLg== --osd-uuid 07250185-8ea5-405f-a4ad-49b8b32635cc 2024-02-05T04:53:29.754 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:29.752+0000 7fa423d074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:29.796 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:29.795+0000 7fa423d074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:29.797 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:29.796+0000 7fa423d074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:29.797 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:29.796+0000 7fa423d074c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:53:29.797 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:29.796+0000 7fa423d074c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:53:29.797 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:29.796+0000 7fa423d074c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:53:29.871 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:29.870+0000 7fa423d074c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:53:35.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T04:53:35.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:53:35.531 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:53:35.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:53:35.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:53:35.878 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:53:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:53:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:53:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:53:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:53:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:53:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:35.905+0000 7f7b53c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:35.909 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:35.908+0000 7f7b53c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:35.910 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:35.909+0000 7f7b53c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:53:36.147 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:53:36.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:36.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:36.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:36.438 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:36.437+0000 7f7b53c204c0 -1 Falling back to public interface 2024-02-05T04:53:37.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:37.413 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:53:37.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:37.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:53:37.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:37.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:37.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:38.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:38.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:38.692 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:53:38.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:53:38.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:38.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:38.761 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:53:38.760+0000 7f7b53c204c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:53:38.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:39.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:39.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:39.970 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:53:39.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:53:39.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:39.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:40.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:41.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:41.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:41.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:53:41.248 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:53:41.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:41.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:41.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:42.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:42.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:42.631 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:53:42.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:53:42.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:42.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:42.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:43.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:43.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:43.908 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:53:43.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:53:43.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:43.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:45.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:45.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:45.188 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:53:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:53:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:45.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:46.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:46.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:46.615 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:53:46.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:53:46.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:46.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:46.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:47.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:47.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:47.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:53:47.892 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:53:47.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:47.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:48.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:49.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:49.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:49.172 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:53:49.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:53:49.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:49.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:49.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:50.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:50.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:50.664 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:53:50.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:53:50.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:50.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:51.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:52.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:52.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:52.101 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:53:52.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:53:52.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:52.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:52.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:53.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:53.486 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:53:53.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:53.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:53:53.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:53.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:54.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:55.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:55.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:55.230 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:53:55.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:53:55.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:55.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:56.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:56.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:56.740 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:53:56.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:53:56.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:56.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:57.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:58.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:58.388 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:53:58.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:53:58.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:58.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:53:58.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:53:59.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:53:59.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:53:59.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:53:59.846 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:53:59.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:53:59.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:54:00.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:01.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:01.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:01.667 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:54:01.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:54:01.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:01.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:54:01.931 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/443842279,v1:127.0.0.1:6811/443842279] [v2:127.0.0.1:6812/443842279,v1:127.0.0.1:6813/443842279] exists,up 07250185-8ea5-405f-a4ad-49b8b32635cc 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:54:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:54:01.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:54:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:54:01.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T04:54:01.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T04:54:01.939 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:54:01.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cfb8fc14-53e2-4205-8626-9a0c8b192c92 2024-02-05T04:54:01.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cfb8fc14-53e2-4205-8626-9a0c8b192c92' 2024-02-05T04:54:01.939 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 cfb8fc14-53e2-4205-8626-9a0c8b192c92 2024-02-05T04:54:01.939 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:54:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpacBl8ODMOBAA7jzzPrQioQrQzVGK8ZGVHg== 2024-02-05T04:54:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpacBl8ODMOBAA7jzzPrQioQrQzVGK8ZGVHg=="}' 2024-02-05T04:54:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cfb8fc14-53e2-4205-8626-9a0c8b192c92 -i td/osd-scrub-repair/2/new.json 2024-02-05T04:54:02.352 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:54:02.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T04:54:02.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDpacBl8ODMOBAA7jzzPrQioQrQzVGK8ZGVHg== --osd-uuid cfb8fc14-53e2-4205-8626-9a0c8b192c92 2024-02-05T04:54:02.395 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:02.393+0000 7f7c7d0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:02.462 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:02.461+0000 7f7c7d0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:02.463 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:02.462+0000 7f7c7d0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:02.463 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:02.462+0000 7f7c7d0164c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:54:02.463 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:02.462+0000 7f7c7d0164c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:54:02.463 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:02.462+0000 7f7c7d0164c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:54:02.537 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:02.536+0000 7f7c7d0164c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T04:54:08.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T04:54:08.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:54:08.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:54:08.661 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:54:08.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:54:09.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:54:09.009 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:54:09.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:54:09.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:54:09.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:54:09.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:54:09.037 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:09.035+0000 7f1a6400e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:09.038 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:09.037+0000 7f1a6400e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:09.039 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:09.038+0000 7f1a6400e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:54:09.276 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:54:09.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:09.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:09.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:09.568 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:09.567+0000 7f1a6400e4c0 -1 Falling back to public interface 2024-02-05T04:54:10.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:10.545 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:54:10.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:10.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:54:10.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:10.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:10.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:11.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:11.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:11.816 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:54:11.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:54:11.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:11.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:12.018 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:12.017+0000 7f1a6400e4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:54:12.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:13.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:13.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:13.111 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:54:13.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:54:13.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:13.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:13.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:14.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:14.433 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:54:14.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:14.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:54:14.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:14.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:14.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:15.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:15.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:15.840 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:54:15.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:54:15.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:15.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:16.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:17.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:17.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:17.118 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:54:17.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:54:17.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:17.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:17.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:18.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:18.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:18.399 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:54:18.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:54:18.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:18.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:18.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:19.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:19.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:19.677 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:54:19.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:54:19.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:19.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:20.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:21.027 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:54:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:54:21.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:21.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:21.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:22.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:22.305 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:54:22.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:22.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:54:22.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:22.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:22.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:23.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:23.584 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:54:23.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:23.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:54:23.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:23.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:24.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:25.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:25.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:25.159 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:54:25.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:54:25.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:25.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:25.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:26.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:26.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:26.927 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:54:26.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:54:26.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:26.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:27.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:28.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:28.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:28.480 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:54:28.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:54:28.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:28.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:28.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:29.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:29.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:29.780 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:54:29.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:54:29.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:29.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:30.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:31.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:31.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:31.652 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:54:31.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:54:31.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:31.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:32.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:33.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:33.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:33.379 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:54:33.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:54:33.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:33.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:33.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:34.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:34.823 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:54:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:54:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:54:35.099 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/365191943,v1:127.0.0.1:6819/365191943] [v2:127.0.0.1:6820/365191943,v1:127.0.0.1:6821/365191943] exists,up cfb8fc14-53e2-4205-8626-9a0c8b192c92 2024-02-05T04:54:35.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:54:35.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:54:35.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:54:35.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-02-05T04:54:35.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T04:54:35.644 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'testpool' created 2024-02-05T04:54:35.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:54:36.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-02-05T04:54:37.035 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 2 2024-02-05T04:54:37.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-02-05T04:54:37.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:54:37.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:54:37.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:54:37.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:54:37.048 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:54:37.048 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:54:37.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:54:37.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:54:37.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:54:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:54:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:54:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:54:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:54:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:54:37.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:54:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:54:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:54:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:54:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:54:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:54:37.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:54:37.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-02-05T04:54:37.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-02-05T04:54:37.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-02-05T04:54:37.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:54:37.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:54:37.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-02-05T04:54:37.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-02-05T04:54:37.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672975' 2024-02-05T04:54:37.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:54:37.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:54:37.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-02-05T04:54:37.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-02-05T04:54:37.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-42949672975 2-64424509448' 2024-02-05T04:54:37.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:54:37.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-02-05T04:54:37.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:54:37.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:54:37.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-02-05T04:54:37.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:54:37.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-02-05T04:54:37.963 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836497 2024-02-05T04:54:37.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-02-05T04:54:37.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:54:38.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-02-05T04:54:38.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:54:39.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:54:39.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:54:39.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836497 2024-02-05T04:54:39.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:54:39.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-02-05T04:54:39.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:54:39.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:54:39.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-02-05T04:54:39.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:54:39.513 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672975 2024-02-05T04:54:39.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-02-05T04:54:39.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-02-05T04:54:39.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:54:39.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-02-05T04:54:39.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:54:39.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-02-05T04:54:39.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:54:39.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:54:39.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-02-05T04:54:39.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:54:39.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-02-05T04:54:39.799 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509448 2024-02-05T04:54:39.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-02-05T04:54:39.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:54:40.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-02-05T04:54:40.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:54:40.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:54:40.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:54:40.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:54:40.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:54:40.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:54:40.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:54:40.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:54:40.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:54:40.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:54:40.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:54:40.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:54:40.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:54:40.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:54:40.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:54:40.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:54:40.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:54:40.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:54:40.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-02-05T04:54:40.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-02-05T04:54:40.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:54:41.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:54:41.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:54:41.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:54:41.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:54:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:54:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:54:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:54:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:54:42.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:54:42.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:54:42.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:54:42.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:54:42.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:54:42.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:54:42.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:54:42.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T04:54:43.291 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T04:54:43.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:54:43.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:54:43.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:54:43.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:54:43.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:54:43.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T04:54:43.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:54:43.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:54:43.812 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:54:43.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:54:43.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:54:43.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:54:43.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:54:43.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:54:43.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:54:43.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:43.840+0000 7f4cf6cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:43.843 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:43.842+0000 7f4cf6cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:43.844 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:43.843+0000 7f4cf6cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:54:44.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:54:44.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:54:44.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:54:44.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:54:44.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:54:44.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:44.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:54:44.086 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:54:44.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:44.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:54:44.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:44.371 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:44.370+0000 7f4cf6cb64c0 -1 Falling back to public interface 2024-02-05T04:54:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:54:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:54:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:45.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:54:45.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:46.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:46.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:46.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:54:46.635 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:54:46.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:46.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:54:46.689 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:54:46.688+0000 7f4cf6cb64c0 -1 osd.0 21 log_to_monitors true 2024-02-05T04:54:46.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:54:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:54:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:54:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:54:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:54:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:54:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:54:48.208 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3454572061,v1:127.0.0.1:6803/3454572061] [v2:127.0.0.1:6804/3454572061,v1:127.0.0.1:6805/3454572061] exists,up 7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584 2024-02-05T04:54:48.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:54:48.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:54:48.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:54:48.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:54:48.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:54:48.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:54:48.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:54:48.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:54:48.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:54:48.563 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:54:48.563 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:54:48.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:54:48.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:54:48.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:54:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T04:54:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T04:54:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T04:54:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:54:48.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:54:48.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-02-05T04:54:48.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-02-05T04:54:48.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978' 2024-02-05T04:54:48.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:54:48.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:54:48.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-02-05T04:54:48.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-02-05T04:54:48.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672978 2-64424509451' 2024-02-05T04:54:48.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:54:48.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T04:54:48.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:54:48.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:54:48.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T04:54:48.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:54:48.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T04:54:48.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T04:54:48.874 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 107374182402 2024-02-05T04:54:48.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:54:49.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T04:54:49.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:54:50.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:54:50.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:54:50.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T04:54:50.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:54:50.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-02-05T04:54:50.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:54:50.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:54:50.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-02-05T04:54:50.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:54:50.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-02-05T04:54:50.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-02-05T04:54:50.420 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672978 2024-02-05T04:54:50.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:54:50.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-02-05T04:54:50.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:54:50.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-02-05T04:54:50.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:54:50.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:54:50.688 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-02-05T04:54:50.688 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:54:50.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-02-05T04:54:50.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-02-05T04:54:50.690 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509451 2024-02-05T04:54:50.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:54:50.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-02-05T04:54:50.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:54:50.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:54:50.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:54:51.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:54:51.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:54:51.289 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:54:51.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:54:51.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:54:51.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:54:51.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:54:51.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:54:51.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:54:51.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:54:51.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:54:51.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:54:51.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:54:51.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:54:51.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:54:51.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-02-05T04:54:51.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T04:54:51.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T04:54:51.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-02-05T04:54:51.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T04:54:52.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-02-05T04:54:52.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-02-05T04:54:52.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:54:52.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:54:52.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:54:52.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:54:52.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-02-05T04:54:52.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-02-05T04:54:52.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:54:52.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:54:52.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:54:52.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:54:52.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:52.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-02-05T04:54:52.822 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T04:54:52.823 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T04:54:52.823 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T04:54:52.823 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T04:53:12.822355+0000" 2024-02-05T04:54:52.823 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T04:54:52.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:52.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T04:54:52.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:52.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T04:54:52.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T04:54:52.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:54:52.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:54:52.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:54:52.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:54:52.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:54:52.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:54:53.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:54:35.594733+0000 '>' 2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:53.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:54:54.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:54:54.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:54:54.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:54:54.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:54:54.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:54:54.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:54:54.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:54:54.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:54:35.594733+0000 '>' 2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:54.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:54:55.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:54:55.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:54:55.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:54:55.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:54:55.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:54:55.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:54:55.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:54:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:54:35.594733+0000 '>' 2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:54:56.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:54:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:54:56.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:54:56.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:54:56.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:54:56.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:54:56.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:54:56.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:54:35.594733+0000 '>' 2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:56.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:54:57.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:54:57.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:54:57.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T04:54:57.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:54:57.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:54:57.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:54:57.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:54:58.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:54:54.127909+0000 '>' 2024-02-05T04:54:35.594733+0000 2024-02-05T04:54:58.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T04:54:58.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T04:54:58.445 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:54:58.445 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T04:54:54.292818+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-02-05T04:54:54.127909+0000 21'1 2024-02-05T04:54:54.127909+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:54:54.127909+0000 1 0 2024-02-05T04:54:58.445 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:54:58.445 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:54:58.445 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:54:58.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-02-05T04:55:00.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T04:55:00.716 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:55:00.716 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T04:54:54.292818+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-02-05T04:54:54.127909+0000 21'1 2024-02-05T04:54:54.127909+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:54:54.127909+0000 1 0 2024-02-05T04:55:00.716 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:55:00.717 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:55:00.717 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:55:00.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-02-05T04:55:02.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T04:55:02.991 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:55:02.991 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T04:54:54.292818+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-02-05T04:54:54.127909+0000 21'1 2024-02-05T04:54:54.127909+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:54:54.127909+0000 1 0 2024-02-05T04:55:02.991 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:55:02.991 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:55:02.991 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:55:03.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-02-05T04:55:08.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-02-05T04:55:08.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:55:08.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:55:08.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:55:08.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:55:08.005 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:55:08.005 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:55:08.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:55:08.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:55:08.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:55:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:55:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:55:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:55:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:55:08.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:55:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:55:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:55:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:55:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:55:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:55:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:55:08.365 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:55:08.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-02-05T04:55:08.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-02-05T04:55:08.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-02-05T04:55:08.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:55:08.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:55:08.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-02-05T04:55:08.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-02-05T04:55:08.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672983' 2024-02-05T04:55:08.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:55:08.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:55:08.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-02-05T04:55:08.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-02-05T04:55:08.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672983 2-64424509456' 2024-02-05T04:55:08.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-02-05T04:55:08.677 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 107374182407 2024-02-05T04:55:08.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:55:08.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182407 2024-02-05T04:55:08.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:55:09.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:55:09.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:55:10.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182407 -lt 107374182407 2024-02-05T04:55:10.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:55:10.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-02-05T04:55:10.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:55:10.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:55:10.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-02-05T04:55:10.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:55:10.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-02-05T04:55:10.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-02-05T04:55:10.226 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672983 2024-02-05T04:55:10.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:55:10.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-02-05T04:55:10.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:55:10.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-02-05T04:55:10.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:55:10.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:55:10.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-02-05T04:55:10.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:55:10.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-02-05T04:55:10.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-02-05T04:55:10.497 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509456 2024-02-05T04:55:10.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:55:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-02-05T04:55:10.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:55:10.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:55:10.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:55:11.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:55:11.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:55:11.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:55:11.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:55:11.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:55:11.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:55:11.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:55:11.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:55:11.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-02-05T04:55:11.994 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:55:11.994 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T04:54:54.292818+0000 21'1 26:62 [1,0] 1 [1,0] 1 21'1 2024-02-05T04:54:54.127909+0000 21'1 2024-02-05T04:54:54.127909+0000 0 1 periodic scrub scheduled @ 2024-02-06T04:54:54.127909+0000 1 0 2024-02-05T04:55:11.994 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:55:11.994 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:55:11.995 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:55:12.006 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:55:12.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:55:12.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:55:12.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:55:12.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:55:12.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:55:12.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:55:12.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:55:12.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T04:55:14.245 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T04:55:14.246 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T04:55:14.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:55:14.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:55:14.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:55:14.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:55:14.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:55:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:55:14.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:55:14.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-02-05T04:55:14.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:55:14.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:55:14.525 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:55:14.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-02-05T04:55:14.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:55:14.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:55:14.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:55:14.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:55:14.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:55:14.553 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:14.552+0000 7f91ae61a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:14.556 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:14.555+0000 7f91ae61a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:14.557 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:14.556+0000 7f91ae61a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:14.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:55:14.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:55:14.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:55:14.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:55:14.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:55:14.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:14.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:55:14.802 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:55:14.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:14.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:15.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:15.080 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:15.079+0000 7f91ae61a4c0 -1 Falling back to public interface 2024-02-05T04:55:16.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:16.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:16.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:55:16.070 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:55:16.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:16.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:16.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:17.314 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:17.313+0000 7f91ae61a4c0 -1 osd.0 26 log_to_monitors true 2024-02-05T04:55:17.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:17.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:17.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:55:17.346 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:55:17.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:17.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:17.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:18.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:18.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:18.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:55:18.654 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:55:18.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:18.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:18.926 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/2524337285,v1:127.0.0.1:6803/2524337285] [v2:127.0.0.1:6804/2524337285,v1:127.0.0.1:6805/2524337285] exists,up 7fa9e5c0-fa1e-4803-ab86-6eccd2ee9584 2024-02-05T04:55:18.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:55:18.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:55:18.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:55:18.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:55:18.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:55:18.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:55:19.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:55:19.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:55:19.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:55:19.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:55:19.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:55:19.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:55:19.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:55:19.273 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:55:19.273 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:55:19.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:55:19.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:55:19.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:55:19.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T04:55:19.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T04:55:19.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-02-05T04:55:19.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:55:19.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:55:19.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-02-05T04:55:19.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-02-05T04:55:19.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986' 2024-02-05T04:55:19.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:55:19.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:55:19.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T04:55:19.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T04:55:19.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672986 2-64424509459' 2024-02-05T04:55:19.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:55:19.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-02-05T04:55:19.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:55:19.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:55:19.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-02-05T04:55:19.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:55:19.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T04:55:19.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-02-05T04:55:19.610 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 128849018882 2024-02-05T04:55:19.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:55:19.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-02-05T04:55:19.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:55:20.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:55:20.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:55:21.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-02-05T04:55:21.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:55:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:55:22.157 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:55:22.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T04:55:22.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:55:22.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-02-05T04:55:22.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:55:22.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:55:22.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-02-05T04:55:22.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:55:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-02-05T04:55:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-02-05T04:55:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672986 2024-02-05T04:55:22.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:55:22.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672986 -lt 42949672986 2024-02-05T04:55:22.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:55:22.703 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T04:55:22.703 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:55:22.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:55:22.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T04:55:22.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:55:22.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T04:55:22.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T04:55:22.706 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509459 2024-02-05T04:55:22.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:55:22.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-02-05T04:55:22.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:55:22.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:55:22.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:55:23.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:55:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:55:23.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:55:23.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:55:23.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:55:23.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:55:23.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:55:23.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:55:23.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T04:55:23.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T04:55:23.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T04:55:23.931 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:53.061+0000 7f7b37664640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-02-05T04:55:23.931 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:53.061+0000 7f7b37664640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-02-05T04:55:23.931 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:53.061+0000 7f7b37664640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-02-05T04:55:23.932 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:54.126+0000 7f7b37664640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:55:23.932 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:54.126+0000 7f7b37664640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T04:55:23.932 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:54.126+0000 7f7b37664640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T04:55:23.932 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:54.126+0000 7f7b37664640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-02-05T04:55:23.932 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:54:54.126+0000 7f7b37664640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:55:23.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-02-05T04:55:23.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:55:24.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:55:24.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:55:24.171 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:55:24.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:55:24.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:55:24.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:55:24.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:55:24.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:55:24.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:55:24.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:55:24.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:55:24.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:55:24.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:55:24.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:55:24.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:55:24.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:55:24.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:24.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:24.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:55:24.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:55:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:55:24.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:55:24.195 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:55:24.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:55:24.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:55:24.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:55:24.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:55:24.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:55:24.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:55:24.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:55:24.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:55:24.199 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:55:24.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:55:24.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:55:24.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:55:24.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T04:55:24.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:55:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:55:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:55:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T04:55:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T04:55:24.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:55:24.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:55:24.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:55:24.598 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:55:24.598 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:24.598 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:24.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:24.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T04:55:24.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:55:24.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:55:24.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:55:24.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:55:24.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T04:55:24.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:55:24.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:55:24.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:55:24.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:55:24.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:55:24.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T04:55:24.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-02-05T04:55:24.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T04:55:24.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:55:24.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:55:24.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:55:24.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T04:55:24.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:55:25.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:55:25.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:55:25.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:55:25.760 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:55:25.760 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:25.760 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:25.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:25.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:55:25.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:55:25.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T04:55:25.787 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-02-05T04:55:25.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-02-05T04:55:25.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:55:25.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:55:25.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:55:25.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:55:25.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:55:25.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:55:25.792 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:55:25.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T04:55:25.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:55:25.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:55:25.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2512f4e8-615c-4435-b9d2-d613fd0c2d9f 2024-02-05T04:55:25.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2512f4e8-615c-4435-b9d2-d613fd0c2d9f' 2024-02-05T04:55:25.795 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 2512f4e8-615c-4435-b9d2-d613fd0c2d9f 2024-02-05T04:55:25.795 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:55:25.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9asBlp1U0MBAA9Bdomfq3e58USIWQjB+/yw== 2024-02-05T04:55:25.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9asBlp1U0MBAA9Bdomfq3e58USIWQjB+/yw=="}' 2024-02-05T04:55:25.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2512f4e8-615c-4435-b9d2-d613fd0c2d9f -i td/osd-scrub-repair/0/new.json 2024-02-05T04:55:26.015 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:55:26.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T04:55:26.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQA9asBlp1U0MBAA9Bdomfq3e58USIWQjB+/yw== --osd-uuid 2512f4e8-615c-4435-b9d2-d613fd0c2d9f 2024-02-05T04:55:26.057 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:26.055+0000 7fc77b6a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:26.066 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:26.065+0000 7fc77b6a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:26.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:26.066+0000 7fc77b6a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:26.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:26.066+0000 7fc77b6a14c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:55:26.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:26.066+0000 7fc77b6a14c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:55:26.067 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:26.066+0000 7fc77b6a14c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:55:26.158 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:26.157+0000 7fc77b6a14c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:55:31.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T04:55:31.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:55:31.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:55:31.633 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:55:31.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:55:31.982 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:55:31.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:55:31.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:55:31.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:55:31.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:55:31.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:55:32.011 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:32.010+0000 7f196ee1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:32.013 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:32.012+0000 7f196ee1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:32.014 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:32.013+0000 7f196ee1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:32.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:55:32.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:55:32.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:55:32.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:55:32.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:55:32.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:32.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:55:32.249 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:55:32.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:32.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:32.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:32.536 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:32.535+0000 7f196ee1a4c0 -1 Falling back to public interface 2024-02-05T04:55:33.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:33.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:33.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:55:33.515 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:55:33.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:33.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:33.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:34.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:34.792 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:55:34.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:34.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:55:34.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:34.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:34.950 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:34.949+0000 7f196ee1a4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:55:35.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:36.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:36.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:36.135 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:55:36.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:55:36.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:36.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:55:36.420 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3052034739,v1:127.0.0.1:6803/3052034739] [v2:127.0.0.1:6804/3052034739,v1:127.0.0.1:6805/3052034739] exists,up 2512f4e8-615c-4435-b9d2-d613fd0c2d9f 2024-02-05T04:55:36.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:55:36.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:55:36.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01781ee5-72db-4b27-a51a-6ced5a2bc241 2024-02-05T04:55:36.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 01781ee5-72db-4b27-a51a-6ced5a2bc241' 2024-02-05T04:55:36.428 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 01781ee5-72db-4b27-a51a-6ced5a2bc241 2024-02-05T04:55:36.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:55:36.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIasBlpYw/GhAAtKG1n6urdU+wFlDtGx+O2w== 2024-02-05T04:55:36.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIasBlpYw/GhAAtKG1n6urdU+wFlDtGx+O2w=="}' 2024-02-05T04:55:36.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01781ee5-72db-4b27-a51a-6ced5a2bc241 -i td/osd-scrub-repair/1/new.json 2024-02-05T04:55:36.779 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:55:36.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T04:55:36.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBIasBlpYw/GhAAtKG1n6urdU+wFlDtGx+O2w== --osd-uuid 01781ee5-72db-4b27-a51a-6ced5a2bc241 2024-02-05T04:55:36.822 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:36.821+0000 7f231d8be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:36.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:36.854+0000 7f231d8be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:36.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:36.855+0000 7f231d8be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:36.857 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:36.856+0000 7f231d8be4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:55:36.857 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:36.856+0000 7f231d8be4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:55:36.857 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:36.856+0000 7f231d8be4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:55:36.933 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:36.932+0000 7f231d8be4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:55:42.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T04:55:42.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:55:42.717 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:55:42.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:55:42.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:55:43.062 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:55:43.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:55:43.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:55:43.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:55:43.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:55:43.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:55:43.090 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:43.089+0000 7f6bd9c164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:43.092 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:43.091+0000 7f6bd9c164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:43.093 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:43.092+0000 7f6bd9c164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:43.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:55:43.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:55:43.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:55:43.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:55:43.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:55:43.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:43.329 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:55:43.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:55:43.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:43.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:55:43.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:43.627 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:43.626+0000 7f6bd9c164c0 -1 Falling back to public interface 2024-02-05T04:55:44.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:44.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:44.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:55:44.598 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:55:44.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:44.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:55:44.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:45.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:45.894 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:55:45.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:45.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:55:45.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:45.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:55:46.047 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:46.046+0000 7f6bd9c164c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:55:46.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:47.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:47.165 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:55:47.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:47.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:55:47.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:47.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:55:47.435 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4205187606,v1:127.0.0.1:6811/4205187606] [v2:127.0.0.1:6812/4205187606,v1:127.0.0.1:6813/4205187606] exists,up 01781ee5-72db-4b27-a51a-6ced5a2bc241 2024-02-05T04:55:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:55:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:55:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:55:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-02-05T04:55:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:55:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:55:47.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:55:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd96fe07-9302-4892-a8a9-003976f54b7d 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fd96fe07-9302-4892-a8a9-003976f54b7d' 2024-02-05T04:55:47.442 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 fd96fe07-9302-4892-a8a9-003976f54b7d 2024-02-05T04:55:47.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:55:47.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTasBljCMdGxAAG8+45BHK4Y6sTOzIzvDsQA== 2024-02-05T04:55:47.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTasBljCMdGxAAG8+45BHK4Y6sTOzIzvDsQA=="}' 2024-02-05T04:55:47.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd96fe07-9302-4892-a8a9-003976f54b7d -i td/osd-scrub-repair/2/new.json 2024-02-05T04:55:47.802 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:55:47.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T04:55:47.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBTasBljCMdGxAAG8+45BHK4Y6sTOzIzvDsQA== --osd-uuid fd96fe07-9302-4892-a8a9-003976f54b7d 2024-02-05T04:55:47.845 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:47.843+0000 7fccbc28c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:47.887 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:47.886+0000 7fccbc28c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:47.888 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:47.887+0000 7fccbc28c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:47.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:47.887+0000 7fccbc28c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:55:47.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:47.887+0000 7fccbc28c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:55:47.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:47.887+0000 7fccbc28c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:55:47.962 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:47.961+0000 7fccbc28c4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T04:55:53.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T04:55:53.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:55:53.853 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:55:53.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:55:53.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:55:54.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:55:54.202 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:55:54.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:55:54.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:55:54.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:55:54.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:55:54.232 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:54.231+0000 7f371761b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:54.234 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:54.233+0000 7f371761b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:54.235 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:54.234+0000 7f371761b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:55:54.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:55:54.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:55:54.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:55:54.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:55:54.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:55:54.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:55:54.470 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:55:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:55:54.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:54.773 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:54.772+0000 7f371761b4c0 -1 Falling back to public interface 2024-02-05T04:55:55.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:55.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:55.738 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:55:55.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:55:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:55:56.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:57.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:57.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:57.019 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:55:57.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:55:57.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:57.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:55:57.166 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:55:57.164+0000 7f371761b4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:55:57.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:55:58.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:55:58.297 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:55:58.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:55:58.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:55:58.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:55:58.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:55:58.584 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3079888983,v1:127.0.0.1:6819/3079888983] [v2:127.0.0.1:6820/3079888983,v1:127.0.0.1:6821/3079888983] exists,up fd96fe07-9302-4892-a8a9-003976f54b7d 2024-02-05T04:55:58.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:55:58.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:55:58.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:55:58.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-02-05T04:55:58.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T04:55:58.958 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'testpool' created 2024-02-05T04:55:58.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:55:59.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-02-05T04:56:00.515 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 2 2024-02-05T04:56:00.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-02-05T04:56:00.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:56:00.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:56:00.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:56:00.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:56:00.529 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:56:00.529 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:56:00.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:56:00.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:56:00.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:56:00.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:56:00.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:56:00.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:56:00.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:56:00.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:56:00.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:56:00.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:56:00.979 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:00.979 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:56:00.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:56:00.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:00.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:56:01.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836487 2024-02-05T04:56:01.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836487 2024-02-05T04:56:01.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487' 2024-02-05T04:56:01.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:01.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:56:01.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T04:56:01.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T04:56:01.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2024-02-05T04:56:01.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:01.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:56:01.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-02-05T04:56:01.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-02-05T04:56:01.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836487 1-42949672964 2-64424509442' 2024-02-05T04:56:01.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:01.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836487 2024-02-05T04:56:01.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:01.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:56:01.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836487 2024-02-05T04:56:01.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:01.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836487 2024-02-05T04:56:01.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-02-05T04:56:01.406 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836487 2024-02-05T04:56:01.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:01.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836486 -lt 21474836487 2024-02-05T04:56:01.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:56:02.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:56:02.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:02.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836487 2024-02-05T04:56:02.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:02.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T04:56:02.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:02.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:56:02.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T04:56:02.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:02.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T04:56:02.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T04:56:02.964 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672964 2024-02-05T04:56:02.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:56:03.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-02-05T04:56:03.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:03.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-02-05T04:56:03.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:03.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:56:03.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-02-05T04:56:03.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:03.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-02-05T04:56:03.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-02-05T04:56:03.233 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509442 2024-02-05T04:56:03.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:56:03.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-02-05T04:56:03.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:56:03.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:03.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:03.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:56:03.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:56:03.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:56:03.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:56:03.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:56:03.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:56:03.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:56:03.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:56:04.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:56:04.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:56:04.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:04.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:04.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:56:04.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:56:04.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:56:04.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-02-05T04:56:04.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-02-05T04:56:04.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:04.567 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:56:04.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:56:04.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:04.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:56:05.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:56:05.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:05.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:05.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:05.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:05.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:05.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T04:56:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:56:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:56:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:56:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:56:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:56:05.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T04:56:06.900 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:56:07.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:56:07.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:56:07.421 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:56:07.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:56:07.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:56:07.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:56:07.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:56:07.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:56:07.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:56:07.451 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:07.450+0000 7fb5aaa154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:07.453 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:07.452+0000 7fb5aaa154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:07.455 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:07.454+0000 7fb5aaa154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:07.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:56:07.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:07.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:56:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:56:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:07.972 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:07.971+0000 7fb5aaa154c0 -1 Falling back to public interface 2024-02-05T04:56:08.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:09.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:09.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:09.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:09.017 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:09.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:09.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:09.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:10.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:10.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:10.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:10.295 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:56:10.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:10.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:10.449 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:10.447+0000 7fb5aaa154c0 -1 osd.0 21 log_to_monitors true 2024-02-05T04:56:10.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:11.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:11.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:11.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:11.655 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:56:11.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:11.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:11.953 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2478076918,v1:127.0.0.1:6803/2478076918] [v2:127.0.0.1:6804/2478076918,v1:127.0.0.1:6805/2478076918] exists,up 2512f4e8-615c-4435-b9d2-d613fd0c2d9f 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:56:11.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:56:11.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:56:12.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:56:12.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:56:12.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:56:12.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:56:12.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:56:12.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:56:12.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:56:12.303 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:12.303 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:56:12.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:56:12.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:12.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:56:12.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-02-05T04:56:12.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-02-05T04:56:12.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-02-05T04:56:12.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:12.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:56:12.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-02-05T04:56:12.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-02-05T04:56:12.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672967' 2024-02-05T04:56:12.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:12.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:56:12.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-02-05T04:56:12.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-02-05T04:56:12.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672967 2-64424509445' 2024-02-05T04:56:12.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:12.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-02-05T04:56:12.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:12.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:56:12.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-02-05T04:56:12.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:12.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-02-05T04:56:12.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-02-05T04:56:12.614 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 107374182402 2024-02-05T04:56:12.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:12.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-02-05T04:56:12.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:56:13.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:56:13.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:14.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-02-05T04:56:14.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:14.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-02-05T04:56:14.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:56:14.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-02-05T04:56:14.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:14.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-02-05T04:56:14.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-02-05T04:56:14.163 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672967 2024-02-05T04:56:14.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:56:14.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-02-05T04:56:14.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:14.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-02-05T04:56:14.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:14.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:56:14.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-02-05T04:56:14.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:14.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-02-05T04:56:14.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-02-05T04:56:14.434 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509445 2024-02-05T04:56:14.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:56:14.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-02-05T04:56:14.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:56:14.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:14.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:56:15.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:56:15.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:56:15.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:56:15.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:56:15.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:56:15.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:56:15.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:56:15.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:56:15.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:56:15.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:15.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:15.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T04:56:16.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-02-05T04:56:16.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-02-05T04:56:16.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:56:16.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:56:16.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:16.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:56:16.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-02-05T04:56:16.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-02-05T04:56:16.279 INFO:tasks.workunit.client.0.smithi188.stdout:Affected PG 1.0 w/ primary 1 2024-02-05T04:56:16.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-02-05T04:56:16.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:56:16.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:56:16.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:56:16.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:56:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-02-05T04:55:58.896425+0000 2024-02-05T04:56:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-02-05T04:56:16.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-02-05T04:56:16.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-02-05T04:56:16.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-02-05T04:56:16.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T04:56:16.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:56:16.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:56:16.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-02-05T04:55:58.896425+0000 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T04:56:16.806 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T04:56:16.807 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:16.807 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:16.807 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:16.807 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T04:56:16.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-02-05T04:56:16.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T04:56:16.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-02-05T04:56:16.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T04:56:16.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-02-05T04:56:16.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:56:16.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:56:17.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:56:17.242 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:17.242 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:56:17.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:56:17.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:17.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:56:17.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182404 2024-02-05T04:56:17.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182404 2024-02-05T04:56:17.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404' 2024-02-05T04:56:17.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:17.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:56:17.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-02-05T04:56:17.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-02-05T04:56:17.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672969' 2024-02-05T04:56:17.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:17.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:56:17.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-05T04:56:17.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-05T04:56:17.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182404 1-42949672969 2-64424509447' 2024-02-05T04:56:17.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:17.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182404 2024-02-05T04:56:17.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:17.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:56:17.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182404 2024-02-05T04:56:17.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:17.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182404 2024-02-05T04:56:17.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182404' 2024-02-05T04:56:17.554 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 107374182404 2024-02-05T04:56:17.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:17.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182404 2024-02-05T04:56:17.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:17.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-02-05T04:56:17.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:17.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:56:17.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-02-05T04:56:17.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:17.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-02-05T04:56:17.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-02-05T04:56:17.824 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672969 2024-02-05T04:56:17.824 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:56:18.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672969 2024-02-05T04:56:18.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:18.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-05T04:56:18.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:18.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:56:18.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-05T04:56:18.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:18.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-05T04:56:18.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-05T04:56:18.133 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509447 2024-02-05T04:56:18.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:56:18.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-05T04:56:18.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-02-05T04:56:18.402 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05 04:56:18.402642664+00:00 2024-02-05T04:56:18.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-02-05T04:56:18.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T04:56:18.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T04:56:18.404 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:18.404 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:18.404 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:18.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T04:56:18.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-02-05T04:56:18.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-02-05T04:56:18.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-02-05T04:56:18.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-02-05T04:56:18.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-02-05T04:56:18.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-02-05T04:56:18.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-02-05T04:56:18.601 INFO:tasks.workunit.client.0.smithi188.stdout:{"deep":true,"must":false,"stamp":"2024-01-22T04:54:38.485364+0000"}0 ) state now: "active+clean" 2024-02-05T04:56:18.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-02-05T04:56:18.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-02-05T04:56:18.601 INFO:tasks.workunit.client.0.smithi188.stdout:"active+clean" 2024-02-05T04:56:18.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-02-05T04:56:18.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-02-05T04:56:18.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-02-05T04:56:18.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-02-05T04:56:18.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-02-05T04:56:18.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-02-05T04:56:19.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-02-05T04:56:19.009 INFO:tasks.workunit.client.0.smithi188.stdout:1 ) state now: "active+clean+scrubbing+deep" 2024-02-05T04:56:19.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T04:56:19.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-02-05T04:56:19.010 INFO:tasks.workunit.client.0.smithi188.stdout:found scrub 2024-02-05T04:56:19.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-02-05T04:56:19.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-02-05T04:56:19.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-02-05T04:56:19.011 INFO:tasks.workunit.client.0.smithi188.stdout:initiate_and_fetch_state ret: 0 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-02-05T04:56:19.013 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:19.014 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:19.014 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:19.014 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.td/osd-scrub-repair.asok 2024-02-05T04:56:19.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-02-05T04:56:19.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T04:56:19.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T04:56:19.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T04:56:19.227 INFO:tasks.workunit.client.0.smithi188.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T04:56:19.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T04:56:19.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T04:56:19.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-02-05T04:56:19.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T04:56:19.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T04:56:19.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T04:56:19.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T04:56:19.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T04:56:19.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T04:56:19.634 INFO:tasks.workunit.client.0.smithi188.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T04:56:19.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T04:56:19.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T04:56:19.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-02-05T04:56:19.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T04:56:19.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T04:56:19.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T04:56:19.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T04:56:19.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T04:56:20.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T04:56:20.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T04:56:20.043 INFO:tasks.workunit.client.0.smithi188.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T04:56:20.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T04:56:20.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-02-05T04:56:20.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T04:56:20.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T04:56:20.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T04:56:20.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T04:56:20.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T04:56:20.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T04:56:20.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T04:56:20.451 INFO:tasks.workunit.client.0.smithi188.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T04:56:20.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T04:56:20.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-02-05T04:56:20.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T04:56:20.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T04:56:20.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T04:56:20.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T04:56:20.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T04:56:20.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-02-05T04:56:20.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-02-05T04:56:20.859 INFO:tasks.workunit.client.0.smithi188.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-02-05T04:56:20.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-02-05T04:56:20.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-02-05T04:56:20.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-02-05T04:56:20.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:56:20.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:56:21.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:56:21.122 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:21.122 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:56:21.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:56:21.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:21.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:56:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182406 2024-02-05T04:56:21.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182406 2024-02-05T04:56:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406' 2024-02-05T04:56:21.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:21.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:56:21.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-02-05T04:56:21.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-02-05T04:56:21.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672971' 2024-02-05T04:56:21.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:21.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:56:21.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T04:56:21.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T04:56:21.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182406 1-42949672971 2-64424509449' 2024-02-05T04:56:21.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:21.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182406 2024-02-05T04:56:21.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:21.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:56:21.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182406 2024-02-05T04:56:21.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:21.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182406 2024-02-05T04:56:21.433 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 107374182406 2024-02-05T04:56:21.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2024-02-05T04:56:21.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:21.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182406 2024-02-05T04:56:21.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:56:22.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:56:22.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:22.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182406 2024-02-05T04:56:22.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:22.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-02-05T04:56:22.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:22.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:56:22.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-02-05T04:56:22.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:22.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-02-05T04:56:22.976 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672971 2024-02-05T04:56:22.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-02-05T04:56:22.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:56:23.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-02-05T04:56:23.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:23.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T04:56:23.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:23.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:56:23.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T04:56:23.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:23.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T04:56:23.245 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509449 2024-02-05T04:56:23.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T04:56:23.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:56:23.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T04:56:23.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-02-05T04:56:23.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-02-05T04:56:23.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-02-05T04:56:23.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-02-05T04:56:23.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-02-05T04:56:23.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-02-05T04:56:23.925 INFO:tasks.workunit.client.0.smithi188.stdout:wait-scrub-end state now: "active+clean" 2024-02-05T04:56:23.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-02-05T04:56:23.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-02-05T04:56:23.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-02-05T04:56:23.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-02-05T04:56:23.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-02-05T04:56:23.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-02-05T04:56:24.190 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T04:56:24.190 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-02-05T04:56:22.123890+0000 21'1 26:51 [1,0] 1 [1,0] 1 21'1 2024-02-05T04:56:21.747314+0000 21'1 2024-02-05T04:56:21.747314+0000 0 4 periodic scrub scheduled @ 2024-02-06T04:56:21.747314+0000 1 0 2024-02-05T04:56:24.190 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T04:56:24.190 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T04:56:24.190 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T04:56:24.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-02-05T04:56:24.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:56:24.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:56:24.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:56:24.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:56:24.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:56:24.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:24.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:56:24.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:56:24.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:24.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:56:24.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:56:24.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:56:24.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:56:24.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:24.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:24.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:24.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:24.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:25.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:25.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T04:56:25.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:56:25.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:56:25.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:56:25.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:56:25.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:56:25.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T04:56:26.354 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T04:56:26.354 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T04:56:26.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:56:26.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:56:26.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:56:26.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:56:26.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:56:26.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:56:26.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:56:26.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:56:26.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:56:26.658 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:26.657+0000 7f4a87c8a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:26.661 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:26.659+0000 7f4a87c8a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:26.662 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:26.660+0000 7f4a87c8a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:26.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:27.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:27.188 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:27.187+0000 7f4a87c8a4c0 -1 Falling back to public interface 2024-02-05T04:56:28.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:28.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:28.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:28.173 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:28.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:28.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:28.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:29.420 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:29.418+0000 7f4a87c8a4c0 -1 osd.0 26 log_to_monitors true 2024-02-05T04:56:29.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:29.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:29.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:29.451 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:56:29.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:29.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:29.843 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/862579828,v1:127.0.0.1:6803/862579828] [v2:127.0.0.1:6804/862579828,v1:127.0.0.1:6805/862579828] exists,up 2512f4e8-615c-4435-b9d2-d613fd0c2d9f 2024-02-05T04:56:29.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:29.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:56:29.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:56:29.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:56:29.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:56:29.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:56:29.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:56:29.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:56:29.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:56:30.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:56:30.274 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:30.274 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:56:30.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:56:30.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:30.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:56:30.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-02-05T04:56:30.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-02-05T04:56:30.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-02-05T04:56:30.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:30.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:56:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-05T04:56:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-05T04:56:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672974' 2024-02-05T04:56:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:30.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:56:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T04:56:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T04:56:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672974 2-64424509452' 2024-02-05T04:56:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:30.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-02-05T04:56:30.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:56:30.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-02-05T04:56:30.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:30.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-02-05T04:56:30.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-02-05T04:56:30.728 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 128849018882 2024-02-05T04:56:30.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:30.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-02-05T04:56:30.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:56:31.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:56:31.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:32.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-02-05T04:56:32.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:32.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-05T04:56:32.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:32.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:56:32.272 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-05T04:56:32.272 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:32.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-05T04:56:32.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-05T04:56:32.273 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672974 2024-02-05T04:56:32.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:56:32.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-02-05T04:56:32.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:32.541 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T04:56:32.541 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:32.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:56:32.542 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T04:56:32.542 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:32.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T04:56:32.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T04:56:32.543 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509452 2024-02-05T04:56:32.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:56:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T04:56:32.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:56:32.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:32.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:33.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:56:33.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:56:33.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:56:33.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:56:33.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:56:33.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:56:33.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:56:33.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:56:33.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:56:33.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:56:33.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:33.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T04:56:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:56:33.746 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:33.746 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:56:34.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T04:56:34.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-02-05T04:56:34.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T04:56:34.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:34.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:34.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:34.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:34.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:56:34.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:56:34.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:56:34.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T04:56:34.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:56:34.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:56:34.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-02-05T04:56:36.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:56:36.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:56:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:56:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T04:56:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:56:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:56:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:56:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:56:36.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:36.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:56:36.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:56:36.289 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:36.287+0000 7fc62b0494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:36.291 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:36.289+0000 7fc62b0494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:36.292 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:36.291+0000 7fc62b0494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:36.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:36.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:36.807 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:36.806+0000 7fc62b0494c0 -1 Falling back to public interface 2024-02-05T04:56:37.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:37.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:37.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:37.802 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:37.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:37.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:38.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:39.059 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:39.057+0000 7fc62b0494c0 -1 osd.0 31 log_to_monitors true 2024-02-05T04:56:39.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:39.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:39.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:39.084 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:56:39.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:39.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:39.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:40.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:40.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:40.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:40.370 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:56:40.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:40.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:40.647 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/674851543,v1:127.0.0.1:6803/674851543] [v2:127.0.0.1:6804/674851543,v1:127.0.0.1:6805/674851543] exists,up 2512f4e8-615c-4435-b9d2-d613fd0c2d9f 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:56:40.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:56:40.649 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:56:40.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:56:40.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:56:40.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:56:40.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:56:40.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:56:40.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:56:40.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:56:40.997 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:56:40.997 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:56:40.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:56:40.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:40.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:56:41.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-02-05T04:56:41.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-02-05T04:56:41.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-02-05T04:56:41.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:41.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:56:41.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-02-05T04:56:41.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-02-05T04:56:41.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672977' 2024-02-05T04:56:41.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:56:41.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:56:41.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-05T04:56:41.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-05T04:56:41.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672977 2-64424509455' 2024-02-05T04:56:41.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:41.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-02-05T04:56:41.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:41.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:56:41.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-02-05T04:56:41.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:41.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-02-05T04:56:41.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-02-05T04:56:41.306 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 150323855362 2024-02-05T04:56:41.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:41.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-02-05T04:56:41.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:56:42.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:56:42.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:56:42.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-02-05T04:56:42.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:42.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-02-05T04:56:42.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:42.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:56:42.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-02-05T04:56:42.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:42.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-02-05T04:56:42.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-02-05T04:56:42.855 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672977 2024-02-05T04:56:42.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:56:43.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-02-05T04:56:43.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:56:43.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-05T04:56:43.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:56:43.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:56:43.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-05T04:56:43.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:56:43.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-05T04:56:43.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-05T04:56:43.127 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509455 2024-02-05T04:56:43.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:56:43.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-05T04:56:43.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:56:43.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:43.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:56:43.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:56:44.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T04:56:44.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:56:44.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:56:44.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:56:44.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T04:56:44.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:56:44.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:56:44.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T04:56:44.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-02-05T04:56:44.342 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:56:21.746+0000 7f6bbfdbc640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-02-05T04:56:44.342 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:56:21.746+0000 7f6bbfdbc640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-02-05T04:56:44.342 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:56:21.746+0000 7f6bbfdbc640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-02-05T04:56:44.343 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:56:21.746+0000 7f6bbfdbc640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-02-05T04:56:44.343 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T04:56:21.746+0000 7f6bbfdbc640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:44.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:44.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:44.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:56:44.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:56:44.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:56:44.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:56:44.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:56:44.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:56:44.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:44.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:56:44.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:56:44.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:44.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:56:44.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:56:44.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:56:44.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:56:44.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:44.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:44.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:56:44.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:56:44.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:56:44.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:56:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:56:44.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:56:44.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:56:44.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:56:44.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:56:44.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:44.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:56:44.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:56:44.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:56:44.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:56:44.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:56:44.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:56:44.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:56:44.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:44.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:44.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:56:44.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:56:44.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:56:44.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:56:44.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:56:44.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:44.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:44.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-02-05T04:56:44.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T04:56:44.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T04:56:44.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:56:44.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:56:44.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:56:44.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T04:56:44.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T04:56:44.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:56:44.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:44.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:44.883 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:44.883 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:44.883 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:44.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:44.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T04:56:45.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:56:45.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:56:45.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:45.052 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:56:45.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:56:45.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T04:56:45.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:56:45.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:56:45.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:56:45.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T04:56:45.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T04:56:45.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T04:56:45.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:56:45.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:56:45.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:56:45.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T04:56:45.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:56:46.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:56:46.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:46.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:46.011 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:46.011 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:46.011 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:46.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:46.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:56:46.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:56:46.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:56:46.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:56:46.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:56:46.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:56:46.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:56:46.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=64e9388a-243a-4b36-8e47-8192df4ede4e 2024-02-05T04:56:46.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 64e9388a-243a-4b36-8e47-8192df4ede4e' 2024-02-05T04:56:46.047 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 64e9388a-243a-4b36-8e47-8192df4ede4e 2024-02-05T04:56:46.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:56:46.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOasBlqMidAxAAmJ9nCfDHBIOAe95hBrXlgA== 2024-02-05T04:56:46.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOasBlqMidAxAAmJ9nCfDHBIOAe95hBrXlgA=="}' 2024-02-05T04:56:46.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 64e9388a-243a-4b36-8e47-8192df4ede4e -i td/osd-scrub-repair/0/new.json 2024-02-05T04:56:46.317 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:56:46.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T04:56:46.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCOasBlqMidAxAAmJ9nCfDHBIOAe95hBrXlgA== --osd-uuid 64e9388a-243a-4b36-8e47-8192df4ede4e 2024-02-05T04:56:46.360 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:46.358+0000 7f880de124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:46.368 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:46.367+0000 7f880de124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:46.369 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:46.368+0000 7f880de124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:46.370 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:46.368+0000 7f880de124c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:56:46.370 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:46.368+0000 7f880de124c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:56:46.370 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:46.368+0000 7f880de124c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:56:46.460 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:46.459+0000 7f880de124c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:56:51.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T04:56:51.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:56:51.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:56:51.891 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:56:51.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:56:52.239 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:56:52.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:56:52.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:56:52.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:56:52.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:56:52.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:56:52.269 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:52.267+0000 7fb431d2b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:52.270 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:52.269+0000 7fb431d2b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:52.271 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:52.270+0000 7fb431d2b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:56:52.512 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:56:52.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:52.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:52.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:52.800 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:52.799+0000 7fb431d2b4c0 -1 Falling back to public interface 2024-02-05T04:56:53.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:53.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:53.779 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:56:53.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:56:53.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:53.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:54.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:55.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:55.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:55.059 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:56:55.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:56:55.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:55.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:55.180 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:56:55.178+0000 7fb431d2b4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:56:55.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:56.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:56.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:56.390 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:56:56.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:56:56.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:56.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:56.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:57.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:57.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:57.672 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:56:57.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:56:57.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:57.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:58.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:56:59.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:56:59.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:56:59.043 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:56:59.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:56:59.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:56:59.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:56:59.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:00.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:00.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:00.321 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:57:00.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:57:00.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:00.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:00.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:01.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:01.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:01.600 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:57:01.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:57:01.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:01.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:01.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:02.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:02.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:02.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:57:02.878 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:57:02.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:02.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:03.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:04.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:04.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:04.228 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:57:04.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:57:04.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:04.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:04.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:05.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:05.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:05.505 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:57:05.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:57:05.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:05.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:05.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:06.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:06.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:06.816 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:57:06.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:57:06.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:06.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:07.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:08.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:08.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:08.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:57:08.555 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:57:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:09.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:10.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:10.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:10.015 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:57:10.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:57:10.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:10.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:10.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:11.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:11.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:11.933 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:57:11.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:57:11.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:11.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:12.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:13.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:13.404 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:57:13.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:13.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:57:13.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:13.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:14.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:15.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:15.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:15.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:57:15.124 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:57:15.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:15.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:15.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:16.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:16.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:16.715 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:57:16.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:57:16.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:16.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:16.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:17.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:17.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:17.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:57:17.997 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:57:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:57:18.268 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2527464053,v1:127.0.0.1:6803/2527464053] [v2:127.0.0.1:6804/2527464053,v1:127.0.0.1:6805/2527464053] exists,up 64e9388a-243a-4b36-8e47-8192df4ede4e 2024-02-05T04:57:18.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:57:18.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:57:18.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:57:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f 2024-02-05T04:57:18.277 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f 2024-02-05T04:57:18.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f' 2024-02-05T04:57:18.278 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:57:18.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuasBl9i4yERAAJIRRGfHaRtqnktcB/wF3Ew== 2024-02-05T04:57:18.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuasBl9i4yERAAJIRRGfHaRtqnktcB/wF3Ew=="}' 2024-02-05T04:57:18.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f -i td/osd-scrub-repair/1/new.json 2024-02-05T04:57:18.613 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:57:18.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T04:57:18.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCuasBl9i4yERAAJIRRGfHaRtqnktcB/wF3Ew== --osd-uuid eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f 2024-02-05T04:57:18.657 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:18.656+0000 7fb9ccb6c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:18.698 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:18.696+0000 7fb9ccb6c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:18.698 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:18.697+0000 7fb9ccb6c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:18.699 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:18.697+0000 7fb9ccb6c4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:57:18.699 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:18.697+0000 7fb9ccb6c4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:57:18.699 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:18.697+0000 7fb9ccb6c4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:57:18.764 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:18.763+0000 7fb9ccb6c4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:57:24.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T04:57:24.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:57:24.784 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:57:24.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:57:24.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:57:25.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:57:25.129 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:57:25.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:57:25.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:57:25.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:57:25.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:57:25.158 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:25.157+0000 7f7e602fa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:25.160 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:25.159+0000 7f7e602fa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:25.161 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:25.160+0000 7f7e602fa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:25.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:57:25.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:57:25.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:57:25.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:57:25.397 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:57:25.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:57:25.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:25.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:57:25.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:25.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:25.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:25.686 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:25.685+0000 7f7e602fa4c0 -1 Falling back to public interface 2024-02-05T04:57:26.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:26.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:26.661 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:57:26.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:57:26.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:26.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:26.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:27.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:27.939 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:57:27.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:27.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:57:27.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:27.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:28.082 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:28.080+0000 7f7e602fa4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:57:28.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:29.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:29.242 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:57:29.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:29.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:57:29.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:29.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:29.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:30.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:30.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:30.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:57:30.563 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:57:30.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:30.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:31.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:32.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:32.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:32.008 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:57:32.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:57:32.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:32.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:32.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:33.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:33.404 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:57:33.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:33.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:57:33.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:33.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:33.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:34.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:34.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:34.716 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:57:34.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:57:34.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:34.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:35.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:36.034 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:57:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:57:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:36.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:37.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:37.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:37.312 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:57:37.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:57:37.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:37.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:37.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:38.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:38.589 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:57:38.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:38.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:57:38.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:38.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:38.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:39.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:39.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:39.976 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:57:39.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:57:39.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:39.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:40.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:41.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:41.328 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:57:41.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:41.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:57:41.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:41.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:41.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:42.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:43.000 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:57:43.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:43.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:57:43.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:43.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:43.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:44.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:44.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:44.720 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:57:44.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:57:44.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:44.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:45.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:46.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:46.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:46.287 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:57:46.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:57:46.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:46.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:46.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:47.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:47.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:47.898 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:57:47.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:57:47.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:47.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:48.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:49.417 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:57:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:57:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:50.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:51.030 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:57:51.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:57:51.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:51.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:51.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:57:52.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:57:52.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:57:52.399 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T04:57:52.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T04:57:52.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:57:52.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:57:52.677 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3165332344,v1:127.0.0.1:6811/3165332344] [v2:127.0.0.1:6812/3165332344,v1:127.0.0.1:6813/3165332344] exists,up eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T04:57:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:57:52.682 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:57:52.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:57:52.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:57:52.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T04:57:52.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:57:52.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86d7d272-b740-4ec1-a6c9-be08ad1937ef 2024-02-05T04:57:52.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 86d7d272-b740-4ec1-a6c9-be08ad1937ef' 2024-02-05T04:57:52.684 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 86d7d272-b740-4ec1-a6c9-be08ad1937ef 2024-02-05T04:57:52.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:57:52.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQasBlEUeGKRAAwDZnToOq5Va6GJ8xZJrq1w== 2024-02-05T04:57:52.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQasBlEUeGKRAAwDZnToOq5Va6GJ8xZJrq1w=="}' 2024-02-05T04:57:52.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86d7d272-b740-4ec1-a6c9-be08ad1937ef -i td/osd-scrub-repair/2/new.json 2024-02-05T04:57:53.050 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:57:53.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T04:57:53.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDQasBlEUeGKRAAwDZnToOq5Va6GJ8xZJrq1w== --osd-uuid 86d7d272-b740-4ec1-a6c9-be08ad1937ef 2024-02-05T04:57:53.093 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:53.092+0000 7fb0175274c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:53.135 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:53.133+0000 7fb0175274c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:53.134+0000 7fb0175274c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:53.135+0000 7fb0175274c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:57:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:53.135+0000 7fb0175274c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:57:53.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:53.135+0000 7fb0175274c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T04:57:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:53.208+0000 7fb0175274c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T04:57:59.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T04:57:59.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:57:59.503 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T04:57:59.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T04:57:59.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:57:59.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T04:57:59.847 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T04:57:59.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:57:59.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:57:59.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:57:59.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:57:59.876 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:59.875+0000 7f12d580c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:59.878 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:59.877+0000 7f12d580c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:57:59.879 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:57:59.878+0000 7f12d580c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:58:00.117 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:58:00.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:00.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:00.408 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:58:00.407+0000 7f12d580c4c0 -1 Falling back to public interface 2024-02-05T04:58:00.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:01.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:01.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:01.430 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:58:01.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:58:01.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:01.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:01.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:02.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:02.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:02.708 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:58:02.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:58:02.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:02.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:02.800 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:58:02.798+0000 7f12d580c4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T04:58:02.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:03.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:03.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:03.985 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:58:03.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:58:03.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:03.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:04.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:05.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:05.375 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:58:05.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:05.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:58:05.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:05.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:05.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:06.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:06.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:06.659 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:58:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:58:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:06.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:07.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:07.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:07.939 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:58:07.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:58:07.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:07.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:08.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:09.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:09.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:09.359 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:58:09.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:58:09.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:09.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:09.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:10.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:10.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:10.636 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:58:10.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:58:10.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:10.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:10.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:11.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:11.916 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:58:11.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:11.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:58:11.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:11.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:12.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:13.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:13.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:13.343 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:58:13.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:58:13.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:13.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:13.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:14.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:14.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:14.744 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:58:14.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:58:14.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:14.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:15.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:16.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:16.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:16.023 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:58:16.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:58:16.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:16.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:16.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:17.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:17.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:17.778 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:58:17.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:58:17.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:17.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:18.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:19.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:19.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:19.056 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:58:19.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:58:19.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:19.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:20.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:20.333 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:58:20.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:20.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:58:20.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:20.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:21.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:22.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:22.219 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:58:22.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:22.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:58:22.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:22.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:22.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:23.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:23.503 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:58:23.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:23.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:58:23.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:23.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:24.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:25.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:25.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:25.186 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:58:25.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:58:25.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:25.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:25.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:26.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:26.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:26.838 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T04:58:26.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T04:58:26.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:26.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T04:58:27.114 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1296174555,v1:127.0.0.1:6819/1296174555] [v2:127.0.0.1:6820/1296174555,v1:127.0.0.1:6821/1296174555] exists,up 86d7d272-b740-4ec1-a6c9-be08ad1937ef 2024-02-05T04:58:27.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:58:27.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:58:27.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:58:27.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-02-05T04:58:27.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T04:58:27.372 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T04:58:27.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T04:58:27.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T04:58:27.849 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T04:58:27.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:58:28.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T04:58:30.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-02-05T04:58:30.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:58:30.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:58:30.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:58:30.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:58:30.818 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:58:30.818 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:58:30.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:58:30.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:58:30.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:58:30.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:58:30.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:58:30.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:58:30.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:58:30.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:58:30.913 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:58:31.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:58:31.179 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:58:31.179 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:58:31.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:58:31.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:31.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:58:31.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-02-05T04:58:31.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-02-05T04:58:31.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-02-05T04:58:31.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:31.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:58:31.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476747 2024-02-05T04:58:31.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476747 2024-02-05T04:58:31.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-68719476747' 2024-02-05T04:58:31.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:31.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:58:31.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920777 2024-02-05T04:58:31.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920777 2024-02-05T04:58:31.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-68719476747 2-141733920777' 2024-02-05T04:58:31.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:31.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-02-05T04:58:31.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:31.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:58:31.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-02-05T04:58:31.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:31.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-02-05T04:58:31.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-02-05T04:58:31.490 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836499 2024-02-05T04:58:31.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:58:31.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-02-05T04:58:31.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:58:32.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:58:32.755 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:58:33.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-02-05T04:58:33.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:33.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476747 2024-02-05T04:58:33.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:33.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:58:33.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476747 2024-02-05T04:58:33.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:33.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476747 2024-02-05T04:58:33.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476747' 2024-02-05T04:58:33.025 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 68719476747 2024-02-05T04:58:33.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:58:33.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476747 -lt 68719476747 2024-02-05T04:58:33.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:33.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920777 2024-02-05T04:58:33.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:33.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:58:33.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920777 2024-02-05T04:58:33.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:33.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920777 2024-02-05T04:58:33.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920777' 2024-02-05T04:58:33.295 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 141733920777 2024-02-05T04:58:33.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:58:33.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920777 -lt 141733920777 2024-02-05T04:58:33.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:58:33.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:58:33.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:58:33.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T04:58:33.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:58:33.950 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:58:33.950 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:58:33.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:58:33.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:58:33.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:58:33.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:58:34.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T04:58:34.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:58:34.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:58:34.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T04:58:34.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T04:58:35.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T04:58:35.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T04:58:35.697 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T04:58:35.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T04:58:36.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T04:58:36.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T04:58:36.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:58:36.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:58:36.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:58:36.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:58:36.711 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:58:36.711 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:58:36.712 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:58:36.712 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:58:36.712 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:58:36.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:58:36.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:58:36.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:58:36.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:58:36.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:58:36.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:58:37.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:58:37.257 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:58:37.257 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:58:37.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:58:37.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:37.257 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:58:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-02-05T04:58:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-02-05T04:58:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-02-05T04:58:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:37.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:58:37.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476749 2024-02-05T04:58:37.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476749 2024-02-05T04:58:37.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-68719476749' 2024-02-05T04:58:37.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:37.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:58:37.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920780 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920780 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-68719476749 2-141733920780' 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-02-05T04:58:37.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:37.746 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836501 2024-02-05T04:58:37.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-02-05T04:58:37.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-02-05T04:58:37.746 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:58:38.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-02-05T04:58:38.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:58:39.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:58:39.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:58:39.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836501 2024-02-05T04:58:39.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:39.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476749 2024-02-05T04:58:39.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:39.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:58:39.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476749 2024-02-05T04:58:39.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:39.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476749 2024-02-05T04:58:39.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476749' 2024-02-05T04:58:39.330 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 68719476749 2024-02-05T04:58:39.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:58:39.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476750 -lt 68719476749 2024-02-05T04:58:39.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:39.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920780 2024-02-05T04:58:39.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:39.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:58:39.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920780 2024-02-05T04:58:39.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920780 2024-02-05T04:58:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920780' 2024-02-05T04:58:39.669 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 141733920780 2024-02-05T04:58:39.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:58:39.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920780 -lt 141733920780 2024-02-05T04:58:39.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:58:39.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:58:39.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:58:40.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:58:40.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T04:58:40.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:58:40.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:58:40.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:58:40.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T04:58:40.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:58:40.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:58:40.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T04:58:40.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-02-05T04:58:40.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-02-05T04:58:40.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T04:58:41.066 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T04:58:41.066 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T04:58:41.066 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:58:41.066 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T04:58:41.066 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T04:58:41.066 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:58:41.066 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T04:58:41.067 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:58:41.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T04:58:41.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T04:58:41.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T04:58:41.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T04:58:41.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:58:41.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:58:41.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:58:41.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:58:41.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T04:58:43.629 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T04:58:43.630 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T04:58:43.630 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:58:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:58:44.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:58:44.150 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T04:58:44.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:58:44.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T04:58:44.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:58:44.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:58:44.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:58:44.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:58:44.181 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:58:44.179+0000 7f5b71a9c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:58:44.197 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:58:44.195+0000 7f5b71a9c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:58:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:58:44.197+0000 7f5b71a9c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:58:44.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:58:44.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:44.717 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:58:44.716+0000 7f5b71a9c4c0 -1 Falling back to public interface 2024-02-05T04:58:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:58:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:58:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:58:45.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:46.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:46.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:46.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:58:46.974 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:58:46.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:46.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:58:47.127 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:58:47.126+0000 7f5b71a9c4c0 -1 osd.1 52 log_to_monitors true 2024-02-05T04:58:47.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:58:48.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:58:48.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:58:48.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:58:48.255 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:58:48.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:58:48.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:58:48.556 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 53 last_clean_interval [16,52) [v2:127.0.0.1:6810/1761731809,v1:127.0.0.1:6811/1761731809] [v2:127.0.0.1:6812/1761731809,v1:127.0.0.1:6813/1761731809] exists,up eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f 2024-02-05T04:58:48.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:58:48.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:58:48.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:58:48.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:58:48.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:58:48.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:58:48.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:58:48.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:58:48.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:58:48.640 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:58:49.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:58:49.009 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:58:49.009 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:58:49.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:58:49.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:49.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:58:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-05T04:58:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-05T04:58:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-05T04:58:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:49.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:58:49.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-02-05T04:58:49.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-02-05T04:58:49.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-249108103170' 2024-02-05T04:58:49.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:49.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:58:49.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920783 2024-02-05T04:58:49.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920783 2024-02-05T04:58:49.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-249108103170 2-141733920783' 2024-02-05T04:58:49.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:49.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-05T04:58:49.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:49.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:58:49.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-05T04:58:49.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:49.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-05T04:58:49.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-05T04:58:49.415 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836505 2024-02-05T04:58:49.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:58:49.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-02-05T04:58:49.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:58:50.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:58:50.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:58:50.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-02-05T04:58:50.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:50.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-02-05T04:58:50.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:50.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:58:50.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-02-05T04:58:50.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:50.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-02-05T04:58:50.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-02-05T04:58:50.964 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 249108103170 2024-02-05T04:58:50.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:58:51.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-02-05T04:58:51.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:51.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920783 2024-02-05T04:58:51.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:51.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:58:51.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920783 2024-02-05T04:58:51.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:51.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920783 2024-02-05T04:58:51.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920783' 2024-02-05T04:58:51.244 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 141733920783 2024-02-05T04:58:51.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:58:51.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920783 -lt 141733920783 2024-02-05T04:58:51.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:58:51.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:58:51.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:58:51.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:58:52.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T04:58:52.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:58:52.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:58:52.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:58:52.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T04:58:52.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:58:52.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:58:52.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T04:58:52.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T04:58:52.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T04:58:52.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T04:58:52.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T04:58:52.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-02-05T04:58:52.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-02-05T04:58:52.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T04:58:52.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:58:52.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:58:52.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-02-05T04:58:41.640563+0000 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T04:58:41.640563+0000 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T04:58:53.105 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T04:58:53.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:58:53.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:58:53.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T04:58:53.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:58:41.640563+0000 '>' 2024-02-05T04:58:41.640563+0000 2024-02-05T04:58:53.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:58:54.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:58:54.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:58:54.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T04:58:54.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T04:58:54.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:58:54.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:58:54.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T04:58:54.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:58:41.640563+0000 '>' 2024-02-05T04:58:41.640563+0000 2024-02-05T04:58:54.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:58:55.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:58:55.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:58:55.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T04:58:55.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T04:58:55.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:58:55.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:58:55.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T04:58:55.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:58:41.640563+0000 '>' 2024-02-05T04:58:41.640563+0000 2024-02-05T04:58:55.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:58:56.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:58:56.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:58:56.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T04:58:56.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T04:58:56.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:58:56.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:58:56.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T04:58:57.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:58:41.640563+0000 '>' 2024-02-05T04:58:41.640563+0000 2024-02-05T04:58:57.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T04:58:58.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T04:58:58.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T04:58:58.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T04:58:58.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T04:58:58.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T04:58:58.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T04:58:58.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T04:58:58.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T04:58:53.724748+0000 '>' 2024-02-05T04:58:41.640563+0000 2024-02-05T04:58:58.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T04:58:58.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-02-05T04:58:58.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:58:58.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:58:58.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:58:58.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:58:58.466 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:58:58.466 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:58:58.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:58:58.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:58:58.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:58:58.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:58:58.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:58:58.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:58:58.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:58:58.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:58:58.549 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:58:58.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:58:58.813 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:58:58.813 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:58:58.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:58:58.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:58.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:58:58.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-05T04:58:58.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-05T04:58:58.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-05T04:58:58.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:58.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:58:59.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103173 2024-02-05T04:58:59.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103173 2024-02-05T04:58:59.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-249108103173' 2024-02-05T04:58:59.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:58:59.020 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920786 2024-02-05T04:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920786 2024-02-05T04:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-249108103173 2-141733920786' 2024-02-05T04:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-05T04:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:58:59.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:58:59.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-05T04:58:59.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:58:59.194 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836508 2024-02-05T04:58:59.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-05T04:58:59.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-05T04:58:59.195 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:58:59.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836508 2024-02-05T04:58:59.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:59:00.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:59:00.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:59:00.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-05T04:59:00.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:59:00.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103173 2024-02-05T04:59:00.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:59:00.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:59:00.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103173 2024-02-05T04:59:00.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:59:00.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103173 2024-02-05T04:59:00.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-02-05T04:59:00.786 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 249108103173 2024-02-05T04:59:00.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:59:01.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103173 -lt 249108103173 2024-02-05T04:59:01.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:59:01.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920786 2024-02-05T04:59:01.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:59:01.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:59:01.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920786 2024-02-05T04:59:01.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:59:01.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920786 2024-02-05T04:59:01.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920786' 2024-02-05T04:59:01.058 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 141733920786 2024-02-05T04:59:01.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:59:01.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920786 -lt 141733920786 2024-02-05T04:59:01.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:59:01.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:59:01.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:59:01.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T04:59:01.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:59:01.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:59:01.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:59:01.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:59:01.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:59:01.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:59:01.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:59:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T04:59:01.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:59:01.939 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:59:01.939 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:59:02.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T04:59:02.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:59:02.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:59:02.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T04:59:02.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T04:59:02.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T04:59:02.289 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T04:59:02.289 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T04:59:02.289 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T04:59:02.289 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T04:59:02.289 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T04:59:02.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T04:59:02.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T04:59:02.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T04:59:02.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:59:02.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:59:02.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:59:02.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:59:02.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:59:03.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:59:03.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T04:59:03.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T04:59:03.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T04:59:03.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T04:59:03.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T04:59:03.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:59:03.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T04:59:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T04:59:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T04:59:04.882 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T04:59:04.882 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T04:59:04.883 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:59:05.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:05.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:59:05.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T04:59:05.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T04:59:05.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T04:59:05.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T04:59:05.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T04:59:05.186 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:05.184+0000 7fb8286104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:05.188 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:05.187+0000 7fb8286104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:05.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:05.188+0000 7fb8286104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T04:59:05.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:05.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:05.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:05.714 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:05.713+0000 7fb8286104c0 -1 Falling back to public interface 2024-02-05T04:59:06.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:06.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:06.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:59:06.706 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:59:06.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:06.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:07.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:08.007 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:08.005+0000 7fb8286104c0 -1 osd.1 65 log_to_monitors true 2024-02-05T04:59:08.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:08.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:08.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:59:08.041 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T04:59:08.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:08.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:08.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:09.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:09.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:09.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:59:09.319 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T04:59:09.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:09.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:09.699 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 73 up_thru 73 down_at 67 last_clean_interval [58,65) [v2:127.0.0.1:6810/587335463,v1:127.0.0.1:6811/587335463] [v2:127.0.0.1:6812/587335463,v1:127.0.0.1:6813/587335463] exists,up eac28fcc-bf6b-4fe2-bbbb-40fb5315e97f 2024-02-05T04:59:09.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:59:09.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:59:09.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T04:59:09.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T04:59:09.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T04:59:09.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T04:59:09.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T04:59:09.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T04:59:09.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T04:59:09.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T04:59:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T04:59:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T04:59:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T04:59:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T04:59:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:59:10.199 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T04:59:10.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-02-05T04:59:10.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-02-05T04:59:10.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-02-05T04:59:10.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:59:10.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T04:59:10.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-02-05T04:59:10.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-02-05T04:59:10.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-313532612610' 2024-02-05T04:59:10.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T04:59:10.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T04:59:10.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920789 2024-02-05T04:59:10.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920789 2024-02-05T04:59:10.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-313532612610 2-141733920789' 2024-02-05T04:59:10.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:59:10.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-02-05T04:59:10.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:59:10.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T04:59:10.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-02-05T04:59:10.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:59:10.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-02-05T04:59:10.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-02-05T04:59:10.524 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836511 2024-02-05T04:59:10.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:59:10.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-02-05T04:59:10.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:59:11.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T04:59:11.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:59:12.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-02-05T04:59:12.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T04:59:13.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T04:59:13.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T04:59:13.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836511 -lt 21474836511 2024-02-05T04:59:13.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:59:13.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-313532612610 2024-02-05T04:59:13.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:59:13.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T04:59:13.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-313532612610 2024-02-05T04:59:13.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:59:13.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-02-05T04:59:13.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 313532612610' 2024-02-05T04:59:13.386 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 313532612610 2024-02-05T04:59:13.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T04:59:13.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-02-05T04:59:13.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T04:59:13.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920789 2024-02-05T04:59:13.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T04:59:13.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T04:59:13.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920789 2024-02-05T04:59:13.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T04:59:13.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920789 2024-02-05T04:59:13.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920789' 2024-02-05T04:59:13.656 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 141733920789 2024-02-05T04:59:13.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T04:59:13.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920789 -lt 141733920789 2024-02-05T04:59:13.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T04:59:13.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:59:13.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T04:59:14.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T04:59:14.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T04:59:14.556 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T04:59:14.556 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T04:59:14.556 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T04:59:14.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T04:59:14.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T04:59:14.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T04:59:14.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T04:59:14.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T04:59:14.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T04:59:14.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:59:14.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:59:14.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:59:14.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:59:14.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:59:14.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:59:14.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:59:14.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:59:15.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:59:15.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:59:15.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:59:15.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:59:15.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:59:15.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:59:15.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:59:15.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:59:15.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:59:15.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:59:15.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:59:15.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:59:15.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:59:15.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:59:15.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:59:15.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:15.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:15.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:59:15.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T04:59:15.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T04:59:15.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T04:59:15.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T04:59:15.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T04:59:15.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T04:59:15.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T04:59:15.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T04:59:15.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T04:59:15.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:59:15.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T04:59:15.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T04:59:15.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T04:59:15.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T04:59:15.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T04:59:15.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T04:59:15.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T04:59:15.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:15.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:15.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T04:59:15.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T04:59:15.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T04:59:15.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T04:59:15.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T04:59:15.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:15.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:15.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T04:59:15.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T04:59:15.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T04:59:15.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T04:59:15.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T04:59:15.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T04:59:15.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:59:15.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:59:15.467 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:59:15.468 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:15.468 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:15.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:15.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T04:59:15.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T04:59:15.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T04:59:15.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:59:15.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:59:15.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T04:59:15.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T04:59:15.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:59:15.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:59:15.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:59:15.654 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:59:15.654 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:15.654 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:15.654 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:59:15.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:59:15.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T04:59:15.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T04:59:15.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T04:59:15.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T04:59:15.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T04:59:15.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T04:59:15.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T04:59:15.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T04:59:15.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T04:59:15.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T04:59:15.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T04:59:15.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T04:59:16.620 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T04:59:16.620 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:59:16.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:59:16.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:59:16.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:16.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:16.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:16.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:59:16.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T04:59:16.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:59:16.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:59:16.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:59:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:59:16.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca32ae58-e478-4109-89ed-781e053a67b9 2024-02-05T04:59:16.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ca32ae58-e478-4109-89ed-781e053a67b9' 2024-02-05T04:59:16.655 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 ca32ae58-e478-4109-89ed-781e053a67b9 2024-02-05T04:59:16.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:59:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAka8BlfAXZJxAAdIu7ZTYWwDAzSxlEpqbxZQ== 2024-02-05T04:59:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAka8BlfAXZJxAAdIu7ZTYWwDAzSxlEpqbxZQ=="}' 2024-02-05T04:59:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca32ae58-e478-4109-89ed-781e053a67b9 -i td/osd-scrub-repair/0/new.json 2024-02-05T04:59:16.851 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:59:16.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T04:59:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAka8BlfAXZJxAAdIu7ZTYWwDAzSxlEpqbxZQ== --osd-uuid ca32ae58-e478-4109-89ed-781e053a67b9 2024-02-05T04:59:16.893 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:16.891+0000 7f30750114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:16.911 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:16.909+0000 7f30750114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:16.912 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:16.910+0000 7f30750114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:16.912 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:16.911+0000 7f30750114c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:59:16.912 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:16.911+0000 7f30750114c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:59:16.912 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:16.911+0000 7f30750114c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T04:59:17.003 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:17.001+0000 7f30750114c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T04:59:22.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T04:59:22.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:59:22.512 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T04:59:22.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T04:59:22.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:59:22.861 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T04:59:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T04:59:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:59:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:59:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:59:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:59:22.890 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:22.888+0000 7fc19900c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:22.892 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:22.891+0000 7fc19900c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:22.893 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:22.892+0000 7fc19900c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:23.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T04:59:23.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:59:23.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T04:59:23.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:59:23.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:59:23.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:23.132 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:59:23.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:59:23.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:23.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:23.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:23.415 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:23.414+0000 7fc19900c4c0 -1 Falling back to public interface 2024-02-05T04:59:24.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:24.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:24.401 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:59:24.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:59:24.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:24.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:24.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:25.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:25.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:25.675 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:59:25.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:59:25.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:25.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:25.765 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:25.763+0000 7fc19900c4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T04:59:25.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:26.950 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:59:26.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:59:26.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:26.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:27.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:28.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:28.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:28.226 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T04:59:28.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T04:59:28.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:28.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:28.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:29.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:29.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:29.587 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T04:59:29.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T04:59:29.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:29.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:29.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:30.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:30.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:30.868 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T04:59:30.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T04:59:30.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:30.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:31.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:32.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:32.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:32.147 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T04:59:32.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T04:59:32.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:32.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:32.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:33.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:33.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:33.427 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T04:59:33.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T04:59:33.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:33.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:33.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:34.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:34.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:34.756 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T04:59:34.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T04:59:34.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:34.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:35.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:36.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:36.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:36.031 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T04:59:36.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T04:59:36.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:36.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:36.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:37.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:37.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:37.309 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T04:59:37.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T04:59:37.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:37.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:37.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:38.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:38.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:38.592 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T04:59:38.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T04:59:38.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:38.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:38.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:39.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:39.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:39.870 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T04:59:39.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T04:59:39.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:39.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:40.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:41.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:41.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:41.260 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T04:59:41.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T04:59:41.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:41.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:41.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:42.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:42.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:42.721 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T04:59:42.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T04:59:42.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:42.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:43.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:44.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:44.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:44.532 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T04:59:44.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T04:59:44.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:44.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:45.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:46.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:46.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:46.301 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T04:59:46.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T04:59:46.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:46.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:47.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:48.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:48.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:48.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T04:59:48.153 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T04:59:48.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:48.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T04:59:48.431 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2886230766,v1:127.0.0.1:6803/2886230766] [v2:127.0.0.1:6804/2886230766,v1:127.0.0.1:6805/2886230766] exists,up ca32ae58-e478-4109-89ed-781e053a67b9 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T04:59:48.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T04:59:48.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T04:59:48.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T04:59:48.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T04:59:48.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T04:59:48.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T04:59:48.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e54735c-ae05-479e-a4d8-fe9fbadd375b 2024-02-05T04:59:48.438 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 3e54735c-ae05-479e-a4d8-fe9fbadd375b 2024-02-05T04:59:48.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3e54735c-ae05-479e-a4d8-fe9fbadd375b' 2024-02-05T04:59:48.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T04:59:48.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEa8BlENbfGhAAeA/Beaky0q9u6n5OjjZwDw== 2024-02-05T04:59:48.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEa8BlENbfGhAAeA/Beaky0q9u6n5OjjZwDw=="}' 2024-02-05T04:59:48.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e54735c-ae05-479e-a4d8-fe9fbadd375b -i td/osd-scrub-repair/1/new.json 2024-02-05T04:59:48.863 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:59:48.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T04:59:48.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBEa8BlENbfGhAAeA/Beaky0q9u6n5OjjZwDw== --osd-uuid 3e54735c-ae05-479e-a4d8-fe9fbadd375b 2024-02-05T04:59:48.906 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:48.904+0000 7f49eac144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:48.939 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:48.938+0000 7f49eac144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:48.940 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:48.939+0000 7f49eac144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:48.941 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:48.939+0000 7f49eac144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:59:48.941 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:48.939+0000 7f49eac144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:59:48.941 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:48.939+0000 7f49eac144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T04:59:49.014 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:49.013+0000 7f49eac144c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T04:59:54.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T04:59:54.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T04:59:54.990 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T04:59:54.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T04:59:54.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T04:59:55.336 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T04:59:55.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T04:59:55.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T04:59:55.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T04:59:55.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T04:59:55.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T04:59:55.365 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:55.363+0000 7f53da0124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:55.367 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:55.365+0000 7f53da0124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:55.368 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:55.366+0000 7f53da0124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T04:59:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T04:59:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T04:59:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T04:59:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T04:59:55.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T04:59:55.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:55.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T04:59:55.609 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T04:59:55.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:55.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:55.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:55.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:55.903+0000 7f53da0124c0 -1 Falling back to public interface 2024-02-05T04:59:56.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:56.875 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T04:59:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T04:59:56.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:56.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:57.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:58.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:58.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:58.154 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T04:59:58.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T04:59:58.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:58.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T04:59:58.250+0000 7f53da0124c0 -1 osd.1 0 log_to_monitors true 2024-02-05T04:59:58.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T04:59:59.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T04:59:59.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T04:59:59.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T04:59:59.462 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T04:59:59.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T04:59:59.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T04:59:59.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:00.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:00.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:00.797 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:00:00.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:00:00.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:00.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:01.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:02.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:02.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:02.180 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:00:02.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:00:02.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:02.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:02.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:03.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:03.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:03.457 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:00:03.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:00:03.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:03.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:03.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:04.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:04.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:04.735 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:00:04.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:00:04.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:04.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:05.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:06.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:06.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:06.066 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:00:06.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:00:06.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:06.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:06.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:07.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:07.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:07.360 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:00:07.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:00:07.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:07.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:07.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:08.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:08.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:08.638 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:00:08.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:00:08.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:08.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:09.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:10.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:10.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:10.017 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:00:10.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:00:10.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:10.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:10.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:11.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:11.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:11.292 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:00:11.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:00:11.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:11.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:11.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:12.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:12.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:12.924 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:00:12.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:00:12.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:12.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:13.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:14.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:14.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:14.306 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:00:14.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:00:14.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:14.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:14.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:15.873 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:00:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:00:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:16.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:17.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:17.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:17.205 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:00:17.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:00:17.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:17.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:17.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:18.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:18.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:18.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:00:18.726 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:00:18.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:18.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:19.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:20.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:20.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:20.521 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:00:20.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:00:20.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:20.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:21.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:22.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:22.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:22.064 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:00:22.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:00:22.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:22.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:00:22.340 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/101513414,v1:127.0.0.1:6811/101513414] [v2:127.0.0.1:6812/101513414,v1:127.0.0.1:6813/101513414] exists,up 3e54735c-ae05-479e-a4d8-fe9fbadd375b 2024-02-05T05:00:22.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:00:22.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:00:22.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:00:22.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:00:22.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:00:22.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cefcf07a-aa94-48b4-b055-f2ab84cda812 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cefcf07a-aa94-48b4-b055-f2ab84cda812' 2024-02-05T05:00:22.347 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 cefcf07a-aa94-48b4-b055-f2ab84cda812 2024-02-05T05:00:22.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:00:22.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBma8BldQ1zFRAAsaMyY6zXTC1731GskRND4w== 2024-02-05T05:00:22.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBma8BldQ1zFRAAsaMyY6zXTC1731GskRND4w=="}' 2024-02-05T05:00:22.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cefcf07a-aa94-48b4-b055-f2ab84cda812 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:00:22.741 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:00:22.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:00:22.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBma8BldQ1zFRAAsaMyY6zXTC1731GskRND4w== --osd-uuid cefcf07a-aa94-48b4-b055-f2ab84cda812 2024-02-05T05:00:22.784 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:22.783+0000 7f4e78c2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:22.818 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:22.816+0000 7f4e78c2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:22.818 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:22.817+0000 7f4e78c2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:22.819 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:22.817+0000 7f4e78c2a4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:00:22.819 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:22.817+0000 7f4e78c2a4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:00:22.819 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:22.817+0000 7f4e78c2a4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:00:22.895 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:22.893+0000 7f4e78c2a4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:00:28.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:00:28.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:00:28.854 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:00:28.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:00:28.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:00:29.201 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:00:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:00:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:00:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:00:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:00:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:00:29.230 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:29.228+0000 7efea0b4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:29.232 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:29.230+0000 7efea0b4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:29.233 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:29.231+0000 7efea0b4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:00:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:00:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:00:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:00:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:00:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:00:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:29.470 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:00:29.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:00:29.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:29.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:29.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:29.761 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:29.760+0000 7efea0b4a4c0 -1 Falling back to public interface 2024-02-05T05:00:30.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:30.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:30.740 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:00:30.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:00:30.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:30.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:31.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:32.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:32.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:32.021 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:00:32.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:00:32.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:32.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:32.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:32.525 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:00:32.523+0000 7efea0b4a4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:00:33.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:33.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:33.318 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:00:33.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:00:33.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:33.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:33.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:34.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:34.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:34.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:00:34.710 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:00:34.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:34.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:34.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:35.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:35.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:35.991 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:00:35.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:00:35.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:35.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:37.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:37.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:37.352 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:00:37.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:00:37.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:37.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:37.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:38.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:38.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:38.652 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:00:38.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:00:38.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:38.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:38.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:39.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:39.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:39.983 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:00:39.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:00:39.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:39.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:40.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:41.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:41.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:41.346 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:00:41.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:00:41.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:41.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:41.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:42.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:42.650 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:00:42.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:42.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:00:42.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:42.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:42.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:43.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:43.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:43.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:00:43.965 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:00:43.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:43.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:44.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:45.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:45.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:45.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:00:45.303 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:00:45.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:45.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:45.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:46.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:46.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:46.723 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:00:46.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:00:46.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:46.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:47.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:48.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:48.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:48.430 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:00:48.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:00:48.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:48.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:49.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:50.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:50.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:50.341 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:00:50.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:00:50.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:50.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:51.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:52.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:52.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:52.153 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:00:52.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:00:52.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:52.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:52.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:53.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:53.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:53.696 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:00:53.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:00:53.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:53.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:54.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:55.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:55.300 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:00:55.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:55.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:00:55.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:55.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:55.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:00:56.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:00:56.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:00:56.794 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:00:56.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:00:56.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:00:56.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:00:57.083 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3605657343,v1:127.0.0.1:6819/3605657343] [v2:127.0.0.1:6820/3605657343,v1:127.0.0.1:6821/3605657343] exists,up cefcf07a-aa94-48b4-b055-f2ab84cda812 2024-02-05T05:00:57.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:00:57.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:00:57.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:00:57.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-02-05T05:00:57.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:00:57.450 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:00:57.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:00:57.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:00:57.927 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:00:57.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:00:58.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:01:01.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-02-05T05:01:01.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:01:01.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:01:01.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:01:01.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:01:01.112 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:01:01.112 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:01:01.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:01:01.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:01:01.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:01:01.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:01:01.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:01:01.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:01:01.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:01:01.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:01:01.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:01:01.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:01:01.516 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:01:01.517 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:01:01.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:01:01.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:01.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:01:01.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T05:01:01.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T05:01:01.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T05:01:01.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:01.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:01:01.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T05:01:01.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T05:01:01.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-64424509452' 2024-02-05T05:01:01.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:01.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:01:01.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920773 2024-02-05T05:01:01.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920773 2024-02-05T05:01:01.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-64424509452 2-141733920773' 2024-02-05T05:01:01.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:01.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T05:01:01.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:01.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:01:01.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T05:01:01.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:01.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T05:01:01.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T05:01:01.855 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836500 2024-02-05T05:01:01.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:02.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-02-05T05:01:02.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:01:03.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:01:03.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:03.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-02-05T05:01:03.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:03.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509452 2024-02-05T05:01:03.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:03.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:01:03.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509452 2024-02-05T05:01:03.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:03.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T05:01:03.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509452' 2024-02-05T05:01:03.414 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 64424509452 2024-02-05T05:01:03.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:01:03.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509452 2024-02-05T05:01:03.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:03.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920773 2024-02-05T05:01:03.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:03.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:01:03.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920773 2024-02-05T05:01:03.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:03.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920773 2024-02-05T05:01:03.686 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 141733920773 2024-02-05T05:01:03.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920773' 2024-02-05T05:01:03.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:01:04.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920774 -lt 141733920773 2024-02-05T05:01:04.013 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:01:04.013 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:04.013 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:01:04.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:01:04.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:01:04.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:01:04.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:04.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:01:04.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T05:01:05.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:01:05.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:01:06.161 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:01:06.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:01:07.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:01:07.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:01:07.897 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T05:01:07.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:01:07.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:01:07.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:01:07.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:01:07.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:01:07.910 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:01:07.910 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:01:07.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:01:07.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:01:07.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:01:07.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:01:07.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:01:07.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:01:07.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:01:07.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:01:07.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:01:08.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:01:08.309 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:01:08.309 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:01:08.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:01:08.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:08.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:01:08.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-02-05T05:01:08.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-02-05T05:01:08.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-02-05T05:01:08.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:08.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:01:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-05T05:01:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-05T05:01:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-64424509455' 2024-02-05T05:01:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:08.556 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:01:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920776 2024-02-05T05:01:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920776 2024-02-05T05:01:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-64424509455 2-141733920776' 2024-02-05T05:01:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:08.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-02-05T05:01:08.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:08.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:01:08.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-02-05T05:01:08.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:08.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-02-05T05:01:08.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-02-05T05:01:08.662 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836503 2024-02-05T05:01:08.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:08.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-02-05T05:01:08.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:08.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509455 2024-02-05T05:01:08.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:08.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:01:08.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509455 2024-02-05T05:01:08.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:08.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-05T05:01:08.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509455' 2024-02-05T05:01:08.931 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 64424509455 2024-02-05T05:01:08.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:01:09.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509455 2024-02-05T05:01:09.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:01:10.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:01:10.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:01:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509455 2024-02-05T05:01:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:01:11.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:01:11.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:01:11.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-05T05:01:11.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:11.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920776 2024-02-05T05:01:11.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:11.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:01:11.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920776 2024-02-05T05:01:11.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:11.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920776 2024-02-05T05:01:11.751 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 141733920776 2024-02-05T05:01:11.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920776' 2024-02-05T05:01:11.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:01:12.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920776 -lt 141733920776 2024-02-05T05:01:12.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:01:12.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:12.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:01:12.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:01:12.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:01:12.718 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:01:12.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:12.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:13.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:01:13.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:01:13.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:01:13.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:01:13.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-02-05T05:01:13.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-02-05T05:01:13.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:01:13.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T05:01:13.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T05:01:13.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:01:13.306 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T05:01:13.306 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:01:13.306 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:01:13.306 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:01:13.307 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:01:13.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T05:01:13.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:01:13.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T05:01:13.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:01:13.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:01:13.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:01:13.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:01:14.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:01:14.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:01:14.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:01:14.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:01:14.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:01:14.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:01:14.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:01:14.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:01:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:01:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T05:01:16.063 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:16.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:16.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:01:16.586 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:01:16.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:01:16.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:01:16.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:01:16.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:01:16.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:01:16.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:01:16.616 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:16.614+0000 7f2a932234c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:16.618 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:16.616+0000 7f2a932234c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:16.619 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:16.618+0000 7f2a932234c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:16.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:01:16.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:16.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:01:16.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:16.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:01:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:17.142 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:17.140+0000 7f2a932234c0 -1 Falling back to public interface 2024-02-05T05:01:17.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:18.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:18.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:18.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:18.165 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:01:18.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:18.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:18.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:19.439 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:19.438+0000 7f2a932234c0 -1 osd.1 58 log_to_monitors true 2024-02-05T05:01:19.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:19.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:19.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:19.441 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:01:19.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:19.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:19.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:20.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:20.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:20.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:01:20.817 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:01:20.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:20.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:21.292 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 59 last_clean_interval [15,58) [v2:127.0.0.1:6810/296040143,v1:127.0.0.1:6811/296040143] [v2:127.0.0.1:6812/296040143,v1:127.0.0.1:6813/296040143] exists,up 3e54735c-ae05-479e-a4d8-fe9fbadd375b 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:01:21.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:01:21.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:01:21.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:01:21.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:01:21.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:01:21.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:01:21.379 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:01:21.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:01:21.771 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:01:21.771 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:01:21.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:01:21.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:21.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:01:21.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-02-05T05:01:21.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-02-05T05:01:21.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-02-05T05:01:21.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:21.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:01:21.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-02-05T05:01:21.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-02-05T05:01:21.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-279172874242' 2024-02-05T05:01:21.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:21.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:01:22.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920780 2024-02-05T05:01:22.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920780 2024-02-05T05:01:22.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-279172874242 2-141733920780' 2024-02-05T05:01:22.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:22.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-02-05T05:01:22.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:22.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:01:22.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-02-05T05:01:22.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:22.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-02-05T05:01:22.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-02-05T05:01:22.086 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836506 2024-02-05T05:01:22.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:22.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836506 2024-02-05T05:01:22.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:01:23.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:01:23.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:23.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-02-05T05:01:23.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:23.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-02-05T05:01:23.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:23.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:01:23.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-02-05T05:01:23.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:23.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-02-05T05:01:23.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-02-05T05:01:23.629 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 279172874242 2024-02-05T05:01:23.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:01:23.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874242 -lt 279172874242 2024-02-05T05:01:23.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:23.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920780 2024-02-05T05:01:23.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:23.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:01:23.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920780 2024-02-05T05:01:23.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:23.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920780 2024-02-05T05:01:23.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920780' 2024-02-05T05:01:23.899 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 141733920780 2024-02-05T05:01:23.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:01:24.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920780 -lt 141733920780 2024-02-05T05:01:24.170 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:01:24.170 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:24.170 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:24.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:01:24.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:01:24.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:01:24.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:01:24.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:01:24.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:01:24.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:01:24.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:01:24.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:01:24.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:01:24.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:24.795 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:25.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:01:25.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:01:25.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:01:25.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:01:25.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:01:25.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-02-05T05:01:25.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-02-05T05:01:25.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:25.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:25.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:25.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:25.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:25.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:25.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:01:12.523012+0000 '>' 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:25.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:01:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:01:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:01:26.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:01:26.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:26.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:26.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:26.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:27.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:01:12.523012+0000 '>' 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:27.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:01:28.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:01:28.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:01:28.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:01:28.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:28.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:28.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:28.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:28.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:01:12.523012+0000 '>' 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:28.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:01:29.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:01:29.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:01:29.532 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:01:29.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:29.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:29.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:29.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:29.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:01:12.523012+0000 '>' 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:29.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:01:30.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:01:30.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:01:30.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:01:30.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:30.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:30.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:30.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:31.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:01:12.523012+0000 '>' 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:31.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:01:32.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:01:32.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:01:32.084 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:01:32.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:32.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:32.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:32.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:32.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:01:12.523012+0000 '>' 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:32.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:01:33.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:01:33.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:01:33.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:01:33.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:01:33.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:01:33.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:01:33.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:01:33.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:01:28.541678+0000 '>' 2024-02-05T05:01:12.523012+0000 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:01:33.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:01:33.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:01:33.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:01:33.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:01:33.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:01:33.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:01:33.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:01:33.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:01:33.999 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:01:33.999 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:01:33.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:01:33.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:33.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:01:34.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-02-05T05:01:34.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-02-05T05:01:34.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-02-05T05:01:34.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:34.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:01:34.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874245 2024-02-05T05:01:34.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874245 2024-02-05T05:01:34.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-279172874245' 2024-02-05T05:01:34.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:34.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:01:34.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920783 2024-02-05T05:01:34.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920783 2024-02-05T05:01:34.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-279172874245 2-141733920783' 2024-02-05T05:01:34.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:34.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-02-05T05:01:34.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:34.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:01:34.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-02-05T05:01:34.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:34.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-02-05T05:01:34.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-02-05T05:01:34.312 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836510 2024-02-05T05:01:34.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:34.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836510 2024-02-05T05:01:34.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:01:35.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:01:35.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:35.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-02-05T05:01:35.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:35.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874245 2024-02-05T05:01:35.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:35.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:01:35.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874245 2024-02-05T05:01:35.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:35.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874245 2024-02-05T05:01:35.876 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 279172874245 2024-02-05T05:01:35.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874245' 2024-02-05T05:01:35.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:01:36.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874245 -lt 279172874245 2024-02-05T05:01:36.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:36.277 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920783 2024-02-05T05:01:36.277 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:36.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:01:36.278 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920783 2024-02-05T05:01:36.278 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:36.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920783 2024-02-05T05:01:36.280 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 141733920783 2024-02-05T05:01:36.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920783' 2024-02-05T05:01:36.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:01:36.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920783 -lt 141733920783 2024-02-05T05:01:36.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:01:36.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:36.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:36.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:01:36.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:01:36.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:01:36.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:01:36.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:01:36.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:01:36.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:01:36.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:01:37.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:01:37.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:01:37.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:37.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:01:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:01:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:01:37.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-02-05T05:01:37.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-02-05T05:01:37.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:01:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-02-05T05:01:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:01:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:01:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:01:37.536 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:01:37.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-02-05T05:01:37.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:01:37.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-02-05T05:01:38.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:01:38.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:01:38.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:01:38.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:01:38.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:01:38.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:01:40.192 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:01:40.192 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-02-05T05:01:40.192 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:01:40.192 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:01:40.192 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:40.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:40.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:01:40.468 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:01:40.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:01:40.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:01:40.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:01:40.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:01:40.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:01:40.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:01:40.499 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:40.497+0000 7faede0244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:40.501 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:40.500+0000 7faede0244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:40.503 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:40.501+0000 7faede0244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:01:40.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:40.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:41.019 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:41.018+0000 7faede0244c0 -1 Falling back to public interface 2024-02-05T05:01:41.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:42.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:01:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:42.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:43.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:43.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:43.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:43.323 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:01:43.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:43.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:43.435 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:43.434+0000 7faede0244c0 -1 osd.1 76 log_to_monitors true 2024-02-05T05:01:43.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:44.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:44.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:44.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:01:44.647 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:01:44.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:44.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:01:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 85 up_thru 85 down_at 79 last_clean_interval [65,76) [v2:127.0.0.1:6810/1716405619,v1:127.0.0.1:6811/1716405619] [v2:127.0.0.1:6812/1716405619,v1:127.0.0.1:6813/1716405619] exists,up 3e54735c-ae05-479e-a4d8-fe9fbadd375b 2024-02-05T05:01:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:01:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:01:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:01:45.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:01:45.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:01:45.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:01:45.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:01:45.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:01:45.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:01:45.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:01:45.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:01:45.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:01:45.581 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:01:45.581 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:01:45.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:01:45.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:45.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:01:45.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-02-05T05:01:45.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-02-05T05:01:45.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-02-05T05:01:45.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:45.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:01:45.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-02-05T05:01:45.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-02-05T05:01:45.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-365072220162' 2024-02-05T05:01:45.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:01:45.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:01:45.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920786 2024-02-05T05:01:45.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920786 2024-02-05T05:01:45.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-365072220162 2-141733920786' 2024-02-05T05:01:45.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:45.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-02-05T05:01:45.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:45.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:01:45.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-02-05T05:01:45.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:45.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-02-05T05:01:45.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-02-05T05:01:45.949 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836513 2024-02-05T05:01:45.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:46.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-02-05T05:01:46.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:01:47.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:01:47.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:01:47.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836513 -lt 21474836513 2024-02-05T05:01:47.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:47.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-365072220162 2024-02-05T05:01:47.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:47.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:01:47.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-365072220162 2024-02-05T05:01:47.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:47.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-02-05T05:01:47.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 365072220162' 2024-02-05T05:01:47.495 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 365072220162 2024-02-05T05:01:47.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:01:47.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220162 -lt 365072220162 2024-02-05T05:01:47.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:01:47.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920786 2024-02-05T05:01:47.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:01:47.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:01:47.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920786 2024-02-05T05:01:47.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:01:47.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920786 2024-02-05T05:01:47.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920786' 2024-02-05T05:01:47.766 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 141733920786 2024-02-05T05:01:47.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:01:48.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920786 -lt 141733920786 2024-02-05T05:01:48.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:01:48.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:48.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:01:48.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:01:48.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:01:48.649 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:01:48.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:01:48.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:01:49.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:01:49.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:01:49.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:01:49.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:01:49.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:01:49.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:01:49.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:01:49.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:01:49.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:01:49.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:01:49.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:01:49.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:01:49.168 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:01:49.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:01:49.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:01:49.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:01:49.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:01:49.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:01:49.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:01:49.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:01:49.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:01:49.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:49.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:49.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:01:49.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:01:49.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:01:49.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:01:49.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:01:49.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:01:49.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:01:49.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:01:49.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:01:49.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:01:49.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:01:49.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:01:49.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:01:49.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:01:49.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:01:49.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:01:49.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:01:49.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:01:49.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:01:49.198 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:01:49.198 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:49.198 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:49.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:01:49.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:01:49.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:01:49.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:01:49.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:01:49.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:49.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:49.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-02-05T05:01:49.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-02-05T05:01:49.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:01:49.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:01:49.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:01:49.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:01:49.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:01:49.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:01:49.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:01:49.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:49.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:49.589 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:49.590 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:49.590 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:49.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:49.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:01:49.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:01:49.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:01:49.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:01:49.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:01:49.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:01:49.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:01:49.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:01:49.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:01:49.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-02-05T05:01:49.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:01:49.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:01:49.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:01:49.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:01:49.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:01:49.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:01:50.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:01:50.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:50.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:50.734 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:50.734 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:50.734 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:50.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:50.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:01:50.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:01:50.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:01:50.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:01:50.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:01:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=465a9cbf-d163-4a0e-b933-7f237028ed81 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 465a9cbf-d163-4a0e-b933-7f237028ed81' 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 465a9cbf-d163-4a0e-b933-7f237028ed81 2024-02-05T05:01:50.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:01:50.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+a8BlA2efLhAAgGur4Q6pPldWtUg8qfpJpw== 2024-02-05T05:01:50.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+a8BlA2efLhAAgGur4Q6pPldWtUg8qfpJpw=="}' 2024-02-05T05:01:50.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 465a9cbf-d163-4a0e-b933-7f237028ed81 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:01:50.990 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:01:51.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:01:51.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+a8BlA2efLhAAgGur4Q6pPldWtUg8qfpJpw== --osd-uuid 465a9cbf-d163-4a0e-b933-7f237028ed81 2024-02-05T05:01:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:51.028+0000 7fad97e254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:51.041 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:51.039+0000 7fad97e254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:51.042 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:51.040+0000 7fad97e254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:51.043 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:51.041+0000 7fad97e254c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:01:51.043 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:51.041+0000 7fad97e254c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:01:51.043 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:51.041+0000 7fad97e254c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:01:51.133 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:51.131+0000 7fad97e254c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:01:56.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:01:56.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:01:56.507 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:01:56.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:01:56.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:01:56.854 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:01:56.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:01:56.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:01:56.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:01:56.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:01:56.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:01:56.880 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:56.878+0000 7fd11bf9b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:56.906 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:56.905+0000 7fd11bf9b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:56.907 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:56.906+0000 7fd11bf9b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:01:57.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:01:57.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:01:57.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:01:57.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:01:57.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:01:57.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:57.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:01:57.128 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:01:57.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:57.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:01:57.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:57.429 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:57.427+0000 7fd11bf9b4c0 -1 Falling back to public interface 2024-02-05T05:01:58.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:58.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:58.413 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:01:58.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:01:58.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:58.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:01:58.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:01:59.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:01:59.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:01:59.688 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:01:59.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:01:59.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:01:59.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:01:59.763 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:01:59.761+0000 7fd11bf9b4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:01:59.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:00.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:00.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:00.958 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:02:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:02:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:01.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:02.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:02.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:02.278 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:02:02.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:02:02.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:02.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:02.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:03.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:03.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:03.557 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:02:03.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:02:03.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:03.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:03.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:04.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:04.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:04.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:02:04.835 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:02:04.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:04.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:05.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:06.136 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:02:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:02:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:06.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:07.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:07.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:07.484 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:02:07.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:02:07.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:07.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:07.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:08.762 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:02:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:02:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:09.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:10.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:10.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:10.043 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:02:10.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:02:10.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:10.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:10.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:11.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:11.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:11.398 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:02:11.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:02:11.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:11.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:11.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:12.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:12.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:12.680 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:02:12.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:02:12.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:12.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:12.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:14.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:14.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:14.000 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:02:14.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:02:14.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:14.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:14.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:15.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:15.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:15.444 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:02:15.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:02:15.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:15.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:16.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:17.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:17.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:17.091 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:02:17.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:02:17.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:17.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:17.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:18.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:18.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:18.791 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:02:18.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:02:18.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:18.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:19.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:20.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:20.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:20.619 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:02:20.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:02:20.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:20.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:21.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:22.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:22.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:22.394 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:02:22.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:02:22.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:22.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:02:22.684 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1680442795,v1:127.0.0.1:6803/1680442795] [v2:127.0.0.1:6804/1680442795,v1:127.0.0.1:6805/1680442795] exists,up 465a9cbf-d163-4a0e-b933-7f237028ed81 2024-02-05T05:02:22.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:02:22.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:02:22.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:02:22.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:02:22.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:02:22.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:02:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b543dab3-3e88-41a2-a182-082434b3e8c1' 2024-02-05T05:02:22.692 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:02:22.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:02:22.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDea8BlwmLvKRAA2GzcwepTNQmPLmhhDMloGw== 2024-02-05T05:02:22.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDea8BlwmLvKRAA2GzcwepTNQmPLmhhDMloGw=="}' 2024-02-05T05:02:22.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b543dab3-3e88-41a2-a182-082434b3e8c1 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:02:23.116 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:02:23.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:02:23.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDea8BlwmLvKRAA2GzcwepTNQmPLmhhDMloGw== --osd-uuid b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:02:23.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:23.155+0000 7f7ae0a104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:23.199+0000 7f7ae0a104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:23.202 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:23.200+0000 7f7ae0a104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:23.202 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:23.201+0000 7f7ae0a104c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:02:23.203 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:23.201+0000 7f7ae0a104c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:02:23.203 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:23.201+0000 7f7ae0a104c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:02:23.276 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:23.274+0000 7f7ae0a104c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:02:28.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:02:28.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:02:28.940 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:02:28.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:02:28.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:02:29.280 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:02:29.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:02:29.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:02:29.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:02:29.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:02:29.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:02:29.306 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:29.304+0000 7f6a7509c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:29.308 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:29.306+0000 7f6a7509c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:29.309 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:29.307+0000 7f6a7509c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:29.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:02:29.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:02:29.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:02:29.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:02:29.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:02:29.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:29.549 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:02:29.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:02:29.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:29.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:29.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:29.846 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:29.844+0000 7f6a7509c4c0 -1 Falling back to public interface 2024-02-05T05:02:30.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:30.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:30.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:02:30.816 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:02:30.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:30.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:31.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:32.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:32.133 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:02:32.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:32.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:02:32.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:32.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:32.167 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:32.165+0000 7f6a7509c4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:02:32.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:33.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:33.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:33.452 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:02:33.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:02:33.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:33.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:33.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:34.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:34.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:34.766 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:02:34.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:02:34.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:34.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:35.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:36.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:36.171 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:02:36.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:36.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:02:36.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:36.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:36.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:37.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:37.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:37.450 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:02:37.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:02:37.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:37.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:37.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:38.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:38.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:38.742 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:02:38.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:02:38.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:38.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:39.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:40.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:40.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:40.062 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:02:40.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:02:40.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:40.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:40.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:41.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:41.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:41.342 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:02:41.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:02:41.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:41.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:41.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:42.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:42.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:42.621 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:02:42.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:02:42.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:42.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:43.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:44.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:44.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:44.062 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:02:44.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:02:44.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:44.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:44.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:45.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:45.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:45.336 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:02:45.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:02:45.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:45.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:46.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:47.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:47.020 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:02:47.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:02:47.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:47.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:47.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:48.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:48.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:48.840 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:02:48.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:02:48.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:48.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:49.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:50.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:50.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:50.550 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:02:50.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:02:50.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:50.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:51.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:52.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:52.327 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:02:52.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:52.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:02:52.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:52.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:52.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:53.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:53.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:53.844 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:02:53.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:02:53.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:53.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:54.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:02:55.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:02:55.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:02:55.484 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:02:55.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:02:55.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:02:55.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:02:55.763 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2602793561,v1:127.0.0.1:6811/2602793561] [v2:127.0.0.1:6812/2602793561,v1:127.0.0.1:6813/2602793561] exists,up b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:02:55.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:02:55.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:02:55.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:02:55.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:02:55.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:02:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:02:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:02:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f035504e-10ad-4285-bcb8-915994b852b7 2024-02-05T05:02:55.771 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 f035504e-10ad-4285-bcb8-915994b852b7 2024-02-05T05:02:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f035504e-10ad-4285-bcb8-915994b852b7' 2024-02-05T05:02:55.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:02:55.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/a8Bl5WWjLhAA0vrzVgcygbEmgQdg55mjTA== 2024-02-05T05:02:55.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/a8Bl5WWjLhAA0vrzVgcygbEmgQdg55mjTA=="}' 2024-02-05T05:02:55.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f035504e-10ad-4285-bcb8-915994b852b7 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:02:56.150 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:02:56.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:02:56.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/a8Bl5WWjLhAA0vrzVgcygbEmgQdg55mjTA== --osd-uuid f035504e-10ad-4285-bcb8-915994b852b7 2024-02-05T05:02:56.191 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:56.188+0000 7f7b8901c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:56.227 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:56.225+0000 7f7b8901c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:56.228 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:56.226+0000 7f7b8901c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:02:56.228 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:56.226+0000 7f7b8901c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:02:56.228 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:56.226+0000 7f7b8901c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:02:56.228 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:56.226+0000 7f7b8901c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:02:56.302 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:02:56.300+0000 7f7b8901c4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:03:02.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:03:02.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:03:02.126 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:03:02.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:03:02.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:03:02.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:03:02.468 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:03:02.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:03:02.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:03:02.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:03:02.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:03:02.494 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:02.492+0000 7fdacdab64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:02.496 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:02.494+0000 7fdacdab64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:02.497 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:02.495+0000 7fdacdab64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:02.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:03:02.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:03:02.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:03:02.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:03:02.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:03:02.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:02.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:03:02.739 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:03:02.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:02.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:03.020 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:03.018+0000 7fdacdab64c0 -1 Falling back to public interface 2024-02-05T05:03:03.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:04.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:04.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:04.031 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:03:04.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:03:04.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:04.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:04.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:05.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:05.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:05.308 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:03:05.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:03:05.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:05.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:05.419 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:05.418+0000 7fdacdab64c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:03:05.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:06.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:06.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:06.620 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:03:06.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:03:06.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:06.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:06.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:07.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:07.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:07.956 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:03:07.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:03:07.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:07.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:08.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:09.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:09.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:09.236 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:03:09.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:03:09.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:09.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:09.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:10.631 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:03:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:03:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:10.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:11.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:11.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:11.910 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:03:11.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:03:11.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:11.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:12.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:13.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:13.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:13.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:03:13.249 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:03:13.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:13.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:13.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:14.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:14.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:14.588 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:03:14.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:03:14.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:14.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:14.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:15.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:15.917 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:03:15.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:15.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:03:15.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:15.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:16.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:17.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:17.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:17.198 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:03:17.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:03:17.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:17.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:17.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:18.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:18.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:18.514 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:03:18.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:03:18.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:18.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:19.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:20.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:20.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:20.231 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:03:20.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:03:20.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:20.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:20.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:21.934 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:03:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:03:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:22.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:23.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:23.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:23.297 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:03:23.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:03:23.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:23.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:24.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:24.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:24.696 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:03:24.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:03:24.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:24.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:25.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:26.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:26.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:26.190 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:03:26.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:03:26.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:26.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:26.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:27.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:27.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:27.541 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:03:27.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:03:27.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:27.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:28.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:29.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:29.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:29.363 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:03:29.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:03:29.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:29.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:03:29.659 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3410570853,v1:127.0.0.1:6819/3410570853] [v2:127.0.0.1:6820/3410570853,v1:127.0.0.1:6821/3410570853] exists,up f035504e-10ad-4285-bcb8-915994b852b7 2024-02-05T05:03:29.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:03:29.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:03:29.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:03:29.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:03:29.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-02-05T05:03:29.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:03:29.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:03:29.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:03:29.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 79318c2b-6f97-4afb-b90f-b6754bde4320' 2024-02-05T05:03:29.666 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:03:29.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:03:29.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhbMBlgtRsKBAA5vnHIG8ta+roR2uYQzMdeA== 2024-02-05T05:03:29.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhbMBlgtRsKBAA5vnHIG8ta+roR2uYQzMdeA=="}' 2024-02-05T05:03:29.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79318c2b-6f97-4afb-b90f-b6754bde4320 -i td/osd-scrub-repair/3/new.json 2024-02-05T05:03:30.040 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:03:30.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:03:30.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhbMBlgtRsKBAA5vnHIG8ta+roR2uYQzMdeA== --osd-uuid 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:03:30.080 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:30.078+0000 7f5bada0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:30.141 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:30.139+0000 7f5bada0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:30.142 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:30.140+0000 7f5bada0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:30.142 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:30.140+0000 7f5bada0e4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:03:30.142 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:30.140+0000 7f5bada0e4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:03:30.142 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:30.140+0000 7f5bada0e4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:03:30.216 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:30.214+0000 7f5bada0e4c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:03:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:03:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:03:36.036 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T05:03:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:03:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:03:36.383 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T05:03:36.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:03:36.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:03:36.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:03:36.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:03:36.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:03:36.409 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:36.406+0000 7f3541f8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:36.411 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:36.408+0000 7f3541f8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:36.412 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:36.410+0000 7f3541f8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:03:36.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:03:36.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:03:36.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:03:36.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:03:36.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:03:36.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:36.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:03:36.654 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:03:36.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:36.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:36.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:36.944 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:36.942+0000 7f3541f8d4c0 -1 Falling back to public interface 2024-02-05T05:03:37.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:37.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:37.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:03:37.939 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:03:37.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:37.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:38.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:39.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:39.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:39.219 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:03:39.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:03:39.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:39.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:39.376 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:03:39.374+0000 7f3541f8d4c0 -1 osd.3 0 log_to_monitors true 2024-02-05T05:03:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:40.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:40.500 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:03:40.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:40.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:03:40.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:40.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:40.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:41.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:41.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:41.799 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:03:41.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:03:41.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:41.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:42.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:43.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:43.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:43.165 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:03:43.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:03:43.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:43.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:43.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:44.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:44.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:44.444 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:03:44.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:03:44.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:44.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:44.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:45.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:45.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:45.786 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:03:45.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:03:45.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:45.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:46.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:47.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:47.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:47.174 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:03:47.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:03:47.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:47.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:47.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:48.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:48.451 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:03:48.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:48.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:03:48.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:48.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:49.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:49.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:49.813 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:03:49.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:03:49.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:49.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:50.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:51.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:51.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:51.195 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:03:51.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:03:51.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:51.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:51.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:52.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:52.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:52.924 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:03:52.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:03:52.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:52.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:53.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:54.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:54.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:54.669 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:03:54.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:03:54.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:54.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:55.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:56.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:56.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:56.113 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:03:56.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:03:56.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:56.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:56.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:57.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:57.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:57.858 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:03:57.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:03:57.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:57.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:03:58.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:03:59.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:03:59.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:03:59.514 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:03:59.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:03:59.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:03:59.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:00.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:01.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:01.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:01.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:04:01.141 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:04:01.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:01.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:01.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:02.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:02.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:02.643 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:04:02.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:04:02.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:02.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:02.930 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/822175367,v1:127.0.0.1:6827/822175367] [v2:127.0.0.1:6828/822175367,v1:127.0.0.1:6829/822175367] exists,up 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:04:02.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:04:02.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:04:02.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:04:02.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-02-05T05:04:02.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:04:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:04:03.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:04:03.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:04:03.572 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:04:03.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:04:04.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:04:07.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-02-05T05:04:07.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:04:07.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:04:07.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:04:07.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:04:07.074 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:04:07.074 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:04:07.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:04:07.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:04:07.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:04:07.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:04:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:04:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:04:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:04:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:04:07.168 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:04:07.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:04:07.493 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:07.493 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:07.493 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:04:07.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:04:07.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:07.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:04:07.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-05T05:04:07.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-05T05:04:07.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-05T05:04:07.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:07.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:04:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-02-05T05:04:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-02-05T05:04:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672978' 2024-02-05T05:04:07.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:07.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:04:07.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T05:04:07.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T05:04:07.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672978 2-64424509449' 2024-02-05T05:04:07.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:07.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:04:07.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345932 2024-02-05T05:04:07.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345932 2024-02-05T05:04:07.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672978 2-64424509449 3-85899345932' 2024-02-05T05:04:07.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:07.904 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-05T05:04:07.904 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:07.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:04:07.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-05T05:04:07.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:07.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-05T05:04:07.907 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836508 2024-02-05T05:04:07.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-05T05:04:07.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:08.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-02-05T05:04:08.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:04:09.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:04:09.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:09.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-05T05:04:09.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:09.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-02-05T05:04:09.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:09.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:04:09.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-02-05T05:04:09.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:09.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-02-05T05:04:09.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-02-05T05:04:09.451 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672978 2024-02-05T05:04:09.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:04:09.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-02-05T05:04:09.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:09.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T05:04:09.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:09.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:04:09.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T05:04:09.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:09.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T05:04:09.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T05:04:09.721 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509449 2024-02-05T05:04:09.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:04:09.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T05:04:09.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:09.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345932 2024-02-05T05:04:09.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:09.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:04:09.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345932 2024-02-05T05:04:09.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:09.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345932 2024-02-05T05:04:09.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-02-05T05:04:09.989 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345932 2024-02-05T05:04:09.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:04:10.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345932 -lt 85899345932 2024-02-05T05:04:10.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:04:10.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:10.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:04:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:04:10.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:04:10.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:04:10.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:04:10.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:04:10.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:04:10.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:04:10.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:04:10.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:04:10.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:10.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:11.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:04:11.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:04:11.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:04:11.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-02-05T05:04:11.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:04:11.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:04:11.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:04:11.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:04:11.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T05:04:11.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:04:11.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:04:12.636 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:04:12.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:04:13.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:04:13.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:04:13.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:04:13.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:04:13.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:04:13.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:04:13.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:04:13.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:04:13.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:04:13.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:04:13.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:04:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:04:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:04:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:04:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:04:13.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:04:13.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:04:14.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:04:14.017 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:14.017 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:14.017 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:04:14.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:04:14.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:14.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:04:14.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836510 2024-02-05T05:04:14.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836510 2024-02-05T05:04:14.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510' 2024-02-05T05:04:14.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:14.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:04:14.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-02-05T05:04:14.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-02-05T05:04:14.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672980' 2024-02-05T05:04:14.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:14.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:04:14.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T05:04:14.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T05:04:14.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672980 2-64424509452' 2024-02-05T05:04:14.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:14.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:04:14.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345934 2024-02-05T05:04:14.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345934 2024-02-05T05:04:14.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836510 1-42949672980 2-64424509452 3-85899345934' 2024-02-05T05:04:14.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:14.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836510 2024-02-05T05:04:14.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:14.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:04:14.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836510 2024-02-05T05:04:14.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:14.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836510 2024-02-05T05:04:14.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-02-05T05:04:14.490 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836510 2024-02-05T05:04:14.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:14.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836510 2024-02-05T05:04:14.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:04:15.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:04:15.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:16.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836510 2024-02-05T05:04:16.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:16.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-02-05T05:04:16.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:16.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:04:16.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-02-05T05:04:16.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-02-05T05:04:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-02-05T05:04:16.033 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672980 2024-02-05T05:04:16.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:04:16.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-02-05T05:04:16.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:16.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T05:04:16.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:16.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:04:16.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T05:04:16.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:16.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T05:04:16.303 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509452 2024-02-05T05:04:16.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T05:04:16.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:04:16.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T05:04:16.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:16.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345934 2024-02-05T05:04:16.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:16.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:04:16.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345934 2024-02-05T05:04:16.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:16.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345934 2024-02-05T05:04:16.574 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345934 2024-02-05T05:04:16.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2024-02-05T05:04:16.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:04:16.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345934 -lt 85899345934 2024-02-05T05:04:16.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:04:16.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:16.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:04:17.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:04:17.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:04:17.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:04:17.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:17.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:04:17.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:04:17.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:04:17.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:04:17.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:04:17.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:04:18.212 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:04:18.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:04:18.589 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:04:18.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:04:18.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:04:18.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:04:18.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:04:18.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:04:18.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:04:18.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:04:18.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:04:19.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:04:19.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:04:19.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:04:19.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:04:19.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:04:19.240 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:04:19.240 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:04:19.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:04:19.510 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:19.511 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:19.511 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:04:19.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-02-05T05:04:19.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-02-05T05:04:19.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:04:19.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-02-05T05:04:19.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:04:19.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:04:19.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:04:19.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:04:19.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:04:19.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:04:19.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:04:19.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:04:21.384 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:04:21.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:04:21.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:04:21.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:04:21.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:04:21.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:04:21.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:04:21.942 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:21.940+0000 7f63d76154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:21.944 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:21.943+0000 7f63d76154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:21.946 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:21.944+0000 7f63d76154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:22.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:22.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:22.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:22.466 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:22.465+0000 7f63d76154c0 -1 Falling back to public interface 2024-02-05T05:04:23.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:23.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:23.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:04:23.459 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:23.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:23.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:23.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:24.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:24.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:24.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:04:24.734 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:24.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:24.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:24.836 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:24.834+0000 7f63d76154c0 -1 osd.3 37 log_to_monitors true 2024-02-05T05:04:25.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:26.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:26.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:26.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:04:26.020 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:04:26.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:26.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:26.396 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [20,37) [v2:127.0.0.1:6826/2951416806,v1:127.0.0.1:6827/2951416806] [v2:127.0.0.1:6828/2951416806,v1:127.0.0.1:6829/2951416806] exists,up 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:04:26.397 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:04:26.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:04:26.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:04:26.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:04:26.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:04:26.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:04:26.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:04:26.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:04:26.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:04:26.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:04:27.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:04:27.016 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:27.016 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:27.016 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:04:27.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:04:27.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:27.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:04:27.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-02-05T05:04:27.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-02-05T05:04:27.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-02-05T05:04:27.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:27.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:04:27.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-02-05T05:04:27.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-02-05T05:04:27.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672984' 2024-02-05T05:04:27.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:27.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:04:27.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-02-05T05:04:27.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-02-05T05:04:27.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672984 2-64424509455' 2024-02-05T05:04:27.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:27.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:04:27.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-02-05T05:04:27.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-02-05T05:04:27.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672984 2-64424509455 3-176093659138' 2024-02-05T05:04:27.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:27.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-02-05T05:04:27.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:27.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:04:27.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-02-05T05:04:27.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:27.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-02-05T05:04:27.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-02-05T05:04:27.459 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836513 2024-02-05T05:04:27.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-02-05T05:04:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:04:28.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:04:28.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:29.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836513 2024-02-05T05:04:29.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:29.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-02-05T05:04:29.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:29.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:04:29.037 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-02-05T05:04:29.037 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:29.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-02-05T05:04:29.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-02-05T05:04:29.039 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672984 2024-02-05T05:04:29.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:04:29.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-02-05T05:04:29.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:29.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-02-05T05:04:29.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:29.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:04:29.307 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-02-05T05:04:29.307 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:29.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-02-05T05:04:29.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-02-05T05:04:29.308 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509455 2024-02-05T05:04:29.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:04:29.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-02-05T05:04:29.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:29.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-176093659138 2024-02-05T05:04:29.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:29.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:04:29.578 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-176093659138 2024-02-05T05:04:29.578 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:29.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-02-05T05:04:29.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 176093659138' 2024-02-05T05:04:29.579 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 176093659138 2024-02-05T05:04:29.579 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:04:29.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-02-05T05:04:29.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:04:29.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:29.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:30.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:04:30.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:04:30.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:04:30.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:04:30.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:04:30.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:04:30.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:04:30.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:04:30.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:04:30.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:04:30.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:30.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:04:30.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:04:31.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:31.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:04:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:04:31.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:04:31.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:04:31.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:12.338025+0000 '>' 2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:31.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:04:32.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:04:32.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:04:32.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:04:32.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:04:32.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:04:32.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:04:32.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:04:33.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:12.338025+0000 '>' 2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:33.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:04:34.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:04:34.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:04:34.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:04:34.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:04:34.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:04:34.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:04:34.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:04:34.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:12.338025+0000 '>' 2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:34.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:04:35.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:04:35.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:04:35.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:04:35.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:04:35.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:04:35.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:04:35.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:04:35.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:12.338025+0000 '>' 2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:35.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:04:36.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:04:36.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:04:36.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:04:36.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:04:36.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:04:36.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:04:36.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:31.982497+0000 '>' 2024-02-05T05:04:12.338025+0000 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:04:36.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:04:36.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:04:37.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:04:37.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:04:37.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:04:37.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:04:37.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:04:37.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:04:37.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:04:37.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:04:39.129 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:04:39.129 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:04:39.130 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:04:39.130 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:04:39.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:04:39.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:04:39.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:04:39.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:04:39.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:04:39.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:04:39.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:04:39.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:04:39.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:04:39.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:04:39.401 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:04:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:04:39.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:04:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:04:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:04:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:04:39.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:04:39.428 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:39.426+0000 7f78f9c164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:39.430 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:39.429+0000 7f78f9c164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:39.431 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:39.430+0000 7f78f9c164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:39.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:39.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:39.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:39.951 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:39.950+0000 7f78f9c164c0 -1 Falling back to public interface 2024-02-05T05:04:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:04:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:41.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:42.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:42.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:42.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:04:42.221 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:42.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:42.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:42.322 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:42.320+0000 7f78f9c164c0 -1 osd.3 43 log_to_monitors true 2024-02-05T05:04:42.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:43.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:43.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:43.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:04:43.528 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:04:43.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:43.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:04:43.933 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [41,43) [v2:127.0.0.1:6826/1631675294,v1:127.0.0.1:6827/1631675294] [v2:127.0.0.1:6828/1631675294,v1:127.0.0.1:6829/1631675294] exists,up 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:04:43.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:04:43.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:04:43.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:04:43.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:04:43.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:04:43.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:04:43.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:04:43.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:04:43.934 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:04:43.934 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:04:43.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:04:43.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:04:43.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:04:44.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:04:44.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:04:44.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:04:44.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:04:44.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:04:44.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:04:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:04:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:44.427 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:04:44.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:04:44.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:44.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:04:44.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836518 2024-02-05T05:04:44.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836518 2024-02-05T05:04:44.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518' 2024-02-05T05:04:44.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:44.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:04:44.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-02-05T05:04:44.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-02-05T05:04:44.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672988' 2024-02-05T05:04:44.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:44.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:04:44.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-02-05T05:04:44.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-02-05T05:04:44.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672988 2-64424509460' 2024-02-05T05:04:44.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:44.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:04:44.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-02-05T05:04:44.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-02-05T05:04:44.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836518 1-42949672988 2-64424509460 3-206158430210' 2024-02-05T05:04:44.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:44.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836518 2024-02-05T05:04:44.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:44.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:04:44.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836518 2024-02-05T05:04:44.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:44.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836518 2024-02-05T05:04:44.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-02-05T05:04:44.841 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836518 2024-02-05T05:04:44.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:45.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836518 2024-02-05T05:04:45.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:45.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-02-05T05:04:45.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:45.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:04:45.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-02-05T05:04:45.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:45.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-02-05T05:04:45.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-02-05T05:04:45.112 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672988 2024-02-05T05:04:45.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:04:45.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-02-05T05:04:45.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:45.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-02-05T05:04:45.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:45.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:04:45.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-02-05T05:04:45.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:45.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-02-05T05:04:45.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-02-05T05:04:45.384 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509460 2024-02-05T05:04:45.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:04:45.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509460 2024-02-05T05:04:45.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:04:46.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:04:46.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:04:46.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-02-05T05:04:46.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:46.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-206158430210 2024-02-05T05:04:46.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:46.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:04:46.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-206158430210 2024-02-05T05:04:46.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:46.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-02-05T05:04:46.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 206158430210' 2024-02-05T05:04:46.936 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 206158430210 2024-02-05T05:04:46.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:04:47.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-02-05T05:04:47.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:04:47.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:47.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:04:47.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:04:47.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:04:47.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:04:47.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:47.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:48.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:04:48.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:04:48.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:04:48.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:04:48.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:04:48.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-02-05T05:04:48.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:04:48.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:04:48.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:04:48.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:04:48.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:04:48.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:04:48.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:04:48.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:04:48.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:04:48.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:04:48.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:04:48.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:04:48.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:04:48.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:04:50.248 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:04:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:04:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:04:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:04:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:04:50.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:04:50.770 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:04:50.771 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:04:50.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:04:50.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:04:50.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:04:50.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:04:50.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:04:50.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:04:50.798 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:50.796+0000 7fb8b94a34c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:50.800 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:50.798+0000 7fb8b94a34c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:50.801 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:50.799+0000 7fb8b94a34c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:51.316 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:51.314+0000 7fb8b94a34c0 -1 Falling back to public interface 2024-02-05T05:04:51.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:52.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:52.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:52.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:04:52.324 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:52.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:52.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:52.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:53.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:53.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:53.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:04:53.603 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:53.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:53.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:53.644 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:04:53.642+0000 7fb8b94a34c0 -1 osd.1 50 log_to_monitors true 2024-02-05T05:04:53.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:04:54.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:04:54.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:04:54.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:04:54.878 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:04:54.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:04:54.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:04:55.182 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [10,50) [v2:127.0.0.1:6810/301828140,v1:127.0.0.1:6811/301828140] [v2:127.0.0.1:6812/301828140,v1:127.0.0.1:6813/301828140] exists,up b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:04:55.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:04:55.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:04:55.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:04:55.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:04:55.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:04:55.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:04:55.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:04:55.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:04:55.183 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:04:55.183 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:04:55.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:04:55.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:04:55.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:04:55.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:04:55.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:04:55.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:04:55.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:04:55.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:04:55.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:04:55.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:04:55.565 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:04:55.565 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:04:55.565 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:04:55.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:04:55.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:55.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:04:55.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836521 2024-02-05T05:04:55.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836521 2024-02-05T05:04:55.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521' 2024-02-05T05:04:55.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:55.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:04:55.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201282 2024-02-05T05:04:55.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201282 2024-02-05T05:04:55.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-236223201282' 2024-02-05T05:04:55.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:55.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:04:56.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-02-05T05:04:56.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-02-05T05:04:56.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-236223201282 2-64424509463' 2024-02-05T05:04:56.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:04:56.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:04:56.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430213 2024-02-05T05:04:56.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430213 2024-02-05T05:04:56.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-236223201282 2-64424509463 3-206158430213' 2024-02-05T05:04:56.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:56.104 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836521 2024-02-05T05:04:56.105 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:56.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:04:56.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836521 2024-02-05T05:04:56.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:56.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836521 2024-02-05T05:04:56.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-02-05T05:04:56.107 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836521 2024-02-05T05:04:56.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:56.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836521 2024-02-05T05:04:56.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:04:57.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:04:57.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:04:57.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836521 2024-02-05T05:04:57.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:57.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201282 2024-02-05T05:04:57.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:57.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:04:57.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201282 2024-02-05T05:04:57.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:57.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201282 2024-02-05T05:04:57.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2024-02-05T05:04:57.653 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 236223201282 2024-02-05T05:04:57.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:04:57.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201282 -lt 236223201282 2024-02-05T05:04:57.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:57.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-02-05T05:04:57.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:57.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:04:57.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-02-05T05:04:57.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:57.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-02-05T05:04:57.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-02-05T05:04:57.924 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509463 2024-02-05T05:04:57.924 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:04:58.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-02-05T05:04:58.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:04:58.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-206158430213 2024-02-05T05:04:58.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:04:58.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:04:58.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-206158430213 2024-02-05T05:04:58.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:04:58.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430213 2024-02-05T05:04:58.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 206158430213' 2024-02-05T05:04:58.193 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 206158430213 2024-02-05T05:04:58.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:04:58.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430213 -lt 206158430213 2024-02-05T05:04:58.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:04:58.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:58.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:58.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:04:58.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:04:58.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:04:58.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:04:58.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:04:58.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:04:58.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:04:58.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:04:59.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:04:59.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:04:59.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:04:59.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:04:59.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:04:59.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:04:59.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:04:59.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:04:59.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:04:59.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:04:59.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:04:59.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:04:59.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:04:59.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:04:59.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:04:59.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:04:59.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:04:59.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:04:59.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:04:59.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:04:59.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:04:31.982497+0000 2024-02-05T05:04:59.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:05:00.175 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:04:31.982497+0000 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:04:31.982497+0000 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:00.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:00.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:31.982497+0000 '>' 2024-02-05T05:04:31.982497+0000 2024-02-05T05:05:00.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:01.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:01.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:01.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:05:01.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:01.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:01.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:01.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:01.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:04:31.982497+0000 '>' 2024-02-05T05:04:31.982497+0000 2024-02-05T05:05:01.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:02.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:02.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:02.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:05:02.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:02.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:02.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:02.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:00.426808+0000 '>' 2024-02-05T05:04:31.982497+0000 2024-02-05T05:05:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:05:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:05:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:05:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:05:03.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:05:04.707 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:05:04.707 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:05:04.707 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:05:04.708 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:05:04.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:05:04.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:04.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:04.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:05:04.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:05:04.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:05:04.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:04.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:04.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:05.006 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:05:05.005+0000 7f6cb080d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:05.009 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:05:05.007+0000 7f6cb080d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:05.010 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:05:05.008+0000 7f6cb080d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:05.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:05.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:05.530 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:05:05.529+0000 7f6cb080d4c0 -1 Falling back to public interface 2024-02-05T05:05:06.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:06.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:06.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:06.525 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:05:06.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:06.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:06.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:07.681 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:05:07.680+0000 7f6cb080d4c0 -1 osd.3 58 log_to_monitors true 2024-02-05T05:05:07.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:07.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:07.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:07.803 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:05:07.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:07.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:08.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:09.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:09.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:09.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:09.071 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:05:09.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:09.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:09.573 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [48,58) [v2:127.0.0.1:6826/1097697423,v1:127.0.0.1:6827/1097697423] [v2:127.0.0.1:6828/1097697423,v1:127.0.0.1:6829/1097697423] exists,up 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:05:09.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:09.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:09.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:09.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:09.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:09.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:09.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:09.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:09.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:09.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:09.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:09.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:09.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:09.994 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:05:09.994 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:05:09.994 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:05:09.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:09.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:09.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:10.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-02-05T05:05:10.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-02-05T05:05:10.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-02-05T05:05:10.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:10.096 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:10.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201286 2024-02-05T05:05:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201286 2024-02-05T05:05:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-236223201286' 2024-02-05T05:05:10.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:10.199 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:05:10.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509467 2024-02-05T05:05:10.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509467 2024-02-05T05:05:10.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-236223201286 2-64424509467' 2024-02-05T05:05:10.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:10.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:05:10.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972354 2024-02-05T05:05:10.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972354 2024-02-05T05:05:10.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-236223201286 2-64424509467 3-266287972354' 2024-02-05T05:05:10.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:10.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-02-05T05:05:10.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:10.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:10.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-02-05T05:05:10.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:10.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-02-05T05:05:10.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-02-05T05:05:10.406 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836525 2024-02-05T05:05:10.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:10.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836525 2024-02-05T05:05:10.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:11.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:11.673 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:11.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-02-05T05:05:11.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:11.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201286 2024-02-05T05:05:11.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:11.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:11.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201286 2024-02-05T05:05:11.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:11.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201286 2024-02-05T05:05:11.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201286' 2024-02-05T05:05:11.952 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 236223201286 2024-02-05T05:05:11.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:12.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201286 -lt 236223201286 2024-02-05T05:05:12.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:12.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509467 2024-02-05T05:05:12.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:12.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:05:12.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509467 2024-02-05T05:05:12.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:12.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509467 2024-02-05T05:05:12.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509467' 2024-02-05T05:05:12.224 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509467 2024-02-05T05:05:12.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:05:12.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509467 -lt 64424509467 2024-02-05T05:05:12.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:12.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972354 2024-02-05T05:05:12.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:12.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:05:12.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972354 2024-02-05T05:05:12.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:12.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972354 2024-02-05T05:05:12.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972354' 2024-02-05T05:05:12.495 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 266287972354 2024-02-05T05:05:12.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:05:12.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972354 -lt 266287972354 2024-02-05T05:05:12.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:05:12.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:12.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:13.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:05:13.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:13.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:13.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:13.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:05:13.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:05:13.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:05:13.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:05:13.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:13.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165125"' 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165125' 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165127"' 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165127' 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 165125 165127' 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:05:13.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165125 2024-02-05T05:05:13.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165128: /' 2024-02-05T05:05:13.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:05:13.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165130: /' 2024-02-05T05:05:16.535 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:05:16.537 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.538 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:16.538 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:05:16.538 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:16.538 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:16.538 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:16.538 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:165128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:16.540 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:05:16.541 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:05:16.541 INFO:tasks.workunit.client.0.smithi188.stderr:165128: start osd.1 2024-02-05T05:05:16.541 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:05:16.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:16.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:16.612 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:05:16.613 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:16.615 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:16.616 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:05:16.616 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:05:16.616 INFO:tasks.workunit.client.0.smithi188.stderr:165130: start osd.2 2024-02-05T05:05:16.616 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:05:21.104 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:05:21.104 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:05:21.104 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 2024-02-05T05:05:16.565+0000 7fae809104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 2024-02-05T05:05:16.568+0000 7fae809104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 2024-02-05T05:05:16.569+0000 7fae809104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 0 2024-02-05T05:05:21.105 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 2024-02-05T05:05:17.084+0000 7fae809104c0 -1 Falling back to public interface 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 1 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:21.106 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 2 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 2024-02-05T05:05:19.744+0000 7fae809104c0 -1 osd.1 64 log_to_monitors true 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 3 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.107 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:21.336 INFO:tasks.workunit.client.0.smithi188.stderr:165128: osd.1 up in weight 1 up_from 70 up_thru 70 down_at 65 last_165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:05:21.336 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:05:21.336 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:21.336 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:21.336 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:21.336 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 2024-02-05T05:05:16.649+0000 7f038a08c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 2024-02-05T05:05:16.652+0000 7f038a08c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 2024-02-05T05:05:16.653+0000 7f038a08c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 0 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 2024-02-05T05:05:17.182+0000 7f038a08c4c0 -1 Falling back to public interface 2024-02-05T05:05:21.337 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 1 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 2 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 2024-02-05T05:05:20.035+0000 7f038a08c4c0 -1 osd.2 64 log_to_monitors true 2024-02-05T05:05:21.339 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:21.339 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:21.339 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:21.339 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 3 2024-02-05T05:05:21.339 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:21.339 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:21.932 INFO:tasks.workunit.client.0.smithi188.stderr:165130: osd.2 up in weight 1 up_from 71 up_thru 51 down_at 65 last_clean_interval [55,64) [v2:127.0.0.1:6810/1305317524,v1:127.0.0.1:6811/1305317524] [v2:127.0.0.1:6812/1305317524,v1:127.0.0.1:6813/1305317524] exists,up b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 1 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 2 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: 3' 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:21.934 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836528 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836528 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528' 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710722 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710722 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-300647710722' 2024-02-05T05:05:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,64) [v2:127.0.0.1:6818/3880518494,v1:127.0.0.1:6819/3880518494] [v2:127.0.0.1:6820/3880518494,v1:127.0.0.1:6821/3880518494] exists,up f035504e-10ad-4285-bcb8-915994b852b7 2024-02-05T05:05:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:22.155 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:22.156 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 1 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 2 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: 3' 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836529 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836529 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529' 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=300647710723 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 300647710723 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-300647710723' 2024-02-05T05:05:22.157 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:24.419 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:05:24.419 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-02-05T05:05:24.419 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-02-05T05:05:24.419 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-300647710722 2-304942678018' 2024-02-05T05:05:24.419 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972357 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972357 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-300647710722 2-304942678018 3-266287972357' 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836528 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836528 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836528 2024-02-05T05:05:24.420 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: waiting osd.0 seq 21474836528 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836528 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836528 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-300647710722 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-300647710722 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710722 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 300647710722' 2024-02-05T05:05:24.421 INFO:tasks.workunit.client.0.smithi188.stderr:165128: waiting osd.1 seq 300647710722 2024-02-05T05:05:24.422 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:24.422 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710723 -lt 300647710722 2024-02-05T05:05:24.422 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:1651tell osd.2 flush_pg_stats 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678019 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678019 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-300647710723 2-304942678019' 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972358 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972358 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-300647710723 2-304942678019 3-266287972358' 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836529 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836529 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836529 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: waiting osd.0 seq 21474836529 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836529 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836529 2024-02-05T05:05:24.610 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-300647710723 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-300647710723 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=300647710723 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 300647710723' 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: waiting osd.1 seq 300647710723 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 300647710723 -lt 300647710723 2024-02-05T05:05:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:25.678 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-304942678018 2024-02-05T05:05:25.678 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:25.678 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:05:25.678 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-304942678018 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: waiting osd.2 seq 304942678018 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678019 -lt 304942678018 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972357 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972357 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972357 2024-02-05T05:05:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972357' 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: waiting osd.3 seq 266287972357 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972359 -lt 266287972357 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:25.681 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:25.681 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:25.681 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:05:25.681 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.c30: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-304942678019 2024-02-05T05:05:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:05:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-304942678019 2024-02-05T05:05:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678019 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 304942678019' 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: waiting osd.2 seq 304942678019 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678019 -lt 304942678019 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972358 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972358 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972358 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972358' 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: waiting osd.3 seq 266287972358 2024-02-05T05:05:25.879 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972359 -lt 266287972358 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:25.880 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:05:25.881 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:27.082 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:27.083 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:05:27.084 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:05:27.085 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:05:27.085 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:05:27.085 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:05:27.085 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:05:27.085 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:05:27.290 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:27.291 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:05:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:05:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:05:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:05:28.342 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:28.343 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:28.344 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:28.345 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:28.345 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:28.345 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:28.345 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:28.345 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:28.565 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/stanclone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:05:28.565 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:28.565 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:28.566 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:28.567 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:28.568 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:29.853 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:29.853 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:29.853 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:05:29.853 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:05:29.853 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:05:29.853 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:29.854 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:165128: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:05:29.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165127 2024-02-05T05:05:30.066 INFO:tasks.workunit.client.0.smithi188.stderr:dalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:30.067 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:30.067 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:05:30.067 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:05:30.067 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:05:30.067 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:05:30.068 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:05:30.068 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:05:30.068 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:05:30.068 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:05:30.068 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:05:30.069 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:05:30.069 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:05:30.069 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:30.069 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:05:30.069 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:165130: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:165130: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:05:30.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:05:30.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:05:30.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:05:30.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:05:30.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:05:30.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:05:30.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:30.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:30.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:30.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:30.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:05:00.426808+0000 2024-02-05T05:05:30.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:05:30.852 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:05:30.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:05:00.426808+0000 2024-02-05T05:05:30.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:05:30.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:05:00.426808+0000 2024-02-05T05:05:30.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:05:30.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:05:30.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:30.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:05:30.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:30.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:30.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:30.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:31.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:00.426808+0000 '>' 2024-02-05T05:05:00.426808+0000 2024-02-05T05:05:31.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:32.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:32.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:32.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:05:32.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:32.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:32.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:32.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:32.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:00.426808+0000 '>' 2024-02-05T05:05:00.426808+0000 2024-02-05T05:05:32.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:33.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:33.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:33.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:05:33.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:33.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:33.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:33.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:33.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:00.426808+0000 '>' 2024-02-05T05:05:00.426808+0000 2024-02-05T05:05:33.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:05:34.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:05:34.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:05:34.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:05:34.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:05:34.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:05:34.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:05:34.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:05:34.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:31.786501+0000 '>' 2024-02-05T05:05:00.426808+0000 2024-02-05T05:05:34.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168441"' 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168441' 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168443"' 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168443' 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:05:34.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:05:34.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 168441 168443' 2024-02-05T05:05:34.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:05:34.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:05:34.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168441 2024-02-05T05:05:34.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168444: /' 2024-02-05T05:05:34.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:05:34.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168446: /' 2024-02-05T05:05:37.379 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.379 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:37.379 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:05:37.379 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:37.379 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:05:37.380 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: _ 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: hinfo_key 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: snapset 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:37.381 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:05:37.382 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:37.382 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:37.382 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:37.382 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:37.382 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:05:37.384 INFO:tasks.workunit.client.0.smithi188.stderr:168446: start osd.2 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:05:37.423 INFO:tasks.workunit.client.0.smithi188.stderr:168444: _ 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: hinfo_key 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: snapset 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:37.425 INFO:tasks.workunit.client.0.smithi188.stderr:168444: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:05:37.426 INFO:tasks.workunit.client.0.smithi188.stderr:168444: start osd.1 2024-02-05T05:05:41.936 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:05:41.936 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:05:41.936 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:05:41.936 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:41.936 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 2024-02-05T05:05:37.451+0000 7fce334144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 2024-02-05T05:05:37.453+0000 7fce334144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 2024-02-05T05:05:37.455+0000 7fce334144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:41.937 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 0 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 2024-02-05T05:05:37.971+0000 7fce334144c0 -1 Falling back to public interface 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 1 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 2 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 2024-02-05T05:05:40.654+0000 7fce334144c0 -1 osd.1 74 log_to_monitors true 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 3 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168444: osd.1 up in weight 1 up_from 80 up_thru 71 down_aovery_ops 2024-02-05T05:05:41.939 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 2024-02-05T05:05:37.409+0000 7f763ca214c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 2024-02-05T05:05:37.412+0000 7f763ca214c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 2024-02-05T05:05:37.413+0000 7f763ca214c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.940 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 0 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 2024-02-05T05:05:37.931+0000 7f763ca214c0 -1 Falling back to public interface 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 1 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.941 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 2 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 2024-02-05T05:05:40.430+0000 7f763ca214c0 -1 osd.2 74 log_to_monitors true 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 3 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:41.942 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:05:42.723 INFO:tasks.workunit.client.0.smithi188.stderr:168446: osd.2 up in weight 1 up_from 79 up_thru 51 down_at 75 last_clean_interval [70,74) [v2:127.0.0.1:6818/2118890535,v1:127.0.0.1:6819/2118890535] [v2:127.0.0.1:6820/2118890535,v1:127.0.0.1:6821/2118890535] exists,up b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:05:42.723 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:42.723 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 1 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 2 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: 3' 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:42.725 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-343597383682' 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 75 last_clean_interval [71,74) [v2:127.0.0.1:6810/1189561379,v1:127.0.0.1:6811/1189561379] [v2:127.0.0.1:6812/1189561379,v1:127.0.0.1:6813/1189561379] exists,up f035504e-10ad-4285-bcb8-915994b852b7 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:42.726 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:42.727 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 1 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 2 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: 3' 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:42.728 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:42.729 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836536 2024-02-05T05:05:42.729 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836536 2024-02-05T05:05:42.729 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536' 2024-02-05T05:05:42.729 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:42.729 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:42.729 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383683 2024-02-05T05:05:42.730 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383683 2024-02-05T05:05:42.730 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-343597383683' 2024-02-05T05:05:42.730 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:44.971 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:05:44.971 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416386 2024-02-05T05:05:44.971 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416386 2024-02-05T05:05:44.971 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-343597383683 2-339302416386' 2024-02-05T05:05:44.971 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:44.971 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:05:44.971 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972364 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972364 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836536 1-343597383683 2-339302416386 3-266287972364' 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836536 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836536 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836536 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836536' 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: waiting osd.0 seq 21474836536 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836536 2024-02-05T05:05:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383683 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383683 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383683 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383683' 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: waiting osd.1 seq 343597383683 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 343597383683 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383683 -lt 343597383683 2024-02-05T05:05:44.974 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:45.084 INFO:tasks.workunit.client.0.smithi188.stderr:168ts: ceph tell osd.2 flush_pg_stats 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416387 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416387 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-343597383682 2-339302416387' 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972365 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972365 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-343597383682 2-339302416387 3-266287972365' 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:45.085 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: waiting osd.0 seq 21474836535 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836535 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-343597383682 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:45.086 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-343597383682 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 343597383682' 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: waiting osd.1 seq 343597383682 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 343597383682 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383683 -lt 343597383682 2024-02-05T05:05:45.087 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:46.225 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416386 2024-02-05T05:05:46.225 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:46.225 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:05:46.225 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416386 2024-02-05T05:05:46.225 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416386 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416386' 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: waiting osd.2 seq 339302416386 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416387 -lt 339302416386 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972364 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972364 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972364 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972364' 2024-02-05T05:05:46.226 INFO:tasks.workunit.client.0.smithi188.stderr:168446: waiting osd.3 seq 266287972364 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972365 -lt 266287972364 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:46.227 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:05:46.228 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:46.308 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416387 2024-02-05T05:05:46.308 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:46.308 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:05:46.308 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416387 2024-02-05T05:05:46.308 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416387 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416387' 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: waiting osd.2 seq 339302416387 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416387 -lt 339302416387 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972365 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972365 2024-02-05T05:05:46.309 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972365 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972365' 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: waiting osd.3 seq 266287972365 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972365 -lt 266287972365 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:46.310 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:46.311 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:46.311 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:46.311 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:46.311 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:46.311 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:46.311 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:05:46.311 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:47.399 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:47.399 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:47.399 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:05:47.399 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:05:47.399 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:47.399 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:47.401 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:47.402 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:47.402 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:47.402 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:47.402 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:47.402 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:47.483 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shclient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:47.483 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:47.483 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:05:47.483 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:05:47.484 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:47.484 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:05:47.484 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:47.484 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:47.484 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:47.484 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:47.484 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:05:47.485 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:47.486 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:48.853 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:48.853 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:48.853 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:48.853 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:48.853 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:48.854 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:48.855 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168446: //home/ubuntu/cephtest/clone.client.0/qa/sta:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:05:49.005 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:05:49.006 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:05:49.007 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:05:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:168444: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:49.229 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:05:49.230 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:05:49.230 INFO:tasks.workunit.client.0.smithi188.stderr:168446: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:05:49.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:ndalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:168444: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168443 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:05:49.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:05:49.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:05:49.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:05:49.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:05:49.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:05:49.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:05:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-02-05T05:05:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:05:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:05:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:05:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-02-05T05:05:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171462"' 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171462' 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171464"' 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171464' 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:05:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171465: /' 2024-02-05T05:05:49.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 171462 171464' 2024-02-05T05:05:49.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:05:49.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:05:49.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171462 2024-02-05T05:05:49.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:49.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171467: /' 2024-02-05T05:05:52.046 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:52.047 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.048 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:52.049 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:05:52.049 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:52.049 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:05:52.049 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:52.049 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:52.050 INFO:tasks.workunit.client.0.smithi188.stderr:171467: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:52.051 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:05:52.052 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:05:52.052 INFO:tasks.workunit.client.0.smithi188.stderr:171467: start osd.1 2024-02-05T05:05:52.052 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:05:52.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:05:52.253 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:05:52.254 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:05:52.255 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:05:52.256 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: start osd.3 2024-02-05T05:05:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 2024-02-05T05:05:52.077+0000 7f92dba154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 2024-02-05T05:05:52.079+0000 7f92dba154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:56.819 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 2024-02-05T05:05:52.080+0000 7f92dba154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 0 2024-02-05T05:05:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 2024-02-05T05:05:52.598+0000 7f92dba154c0 -1 Falling back to public interface 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 1 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 2 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 2024-02-05T05:05:55.191+0000 7f92dba154c0 -1 osd.1 82 log_to_monitors true 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.822 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 3 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171467: osd.1 up in weight 1 up_from 87 up_thru 87 down_at 83 last_171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 2024-02-05T05:05:52.278+0000 7fe0e61fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 2024-02-05T05:05:52.280+0000 7fe0e61fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:56.823 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 2024-02-05T05:05:52.282+0000 7fe0e61fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 0 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:56.824 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 2024-02-05T05:05:52.799+0000 7fe0e61fb4c0 -1 Falling back to public interface 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 1 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:05:56.825 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 2 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 2024-02-05T05:05:55.584+0000 7fe0e61fb4c0 -1 osd.3 82 log_to_monitors true 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 3 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:05:56.826 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:05:57.601 INFO:tasks.workunit.client.0.smithi188.stderr:171465: osd.3 up in weight 1 up_from 89 up_thru 80 down_at 83 last_clean_interval [80,82) [v2:127.0.0.1:6818/2613807676,v1:127.0.0.1:6819/2613807676] [v2:127.0.0.1:6820/2613807676,v1:127.0.0.1:6821/2613807676] exists,up b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:05:57.601 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:57.601 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:57.602 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 1 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 2 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: 3' 2024-02-05T05:05:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-02-05T05:05:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-02-05T05:05:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-02-05T05:05:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154755 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154755 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-373662154755' 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [62,82) [v2:127.0.0.1:6826/1691605109,v1:127.0.0.1:6827/1691605109] [v2:127.0.0.1:6828/1691605109,v1:127.0.0.1:6829/1691605109] exists,up 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:05:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:05:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:05:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:05:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:171465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:05:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:171465: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:05:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:05:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:05:57.606 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 1 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 2 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: 3' 2024-02-05T05:05:57.607 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:05:57.608 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154754 2024-02-05T05:05:57.609 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154754 2024-02-05T05:05:57.609 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-373662154754' 2024-02-05T05:05:57.609 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416392 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416392 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-373662154755 2-339302416392' 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-373662154755 2-339302416392 3-382252089346' 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-02-05T05:05:59.706 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: waiting osd.0 seq 21474836540 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836540 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836540 2024-02-05T05:05:59.707 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154755 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154755 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154755 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154755' 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: waiting osd.1 seq 373662154755 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154755 -lt 373662154755 2024-02-05T05:05:59.708 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:1714tell osd.2 flush_pg_stats 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416391 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416391 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-373662154754 2-339302416391' 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089347 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089347 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-373662154754 2-339302416391 3-382252089347' 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:59.709 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: waiting osd.0 seq 21474836541 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836541 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154754 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:05:59.710 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154754 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154754 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: waiting osd.1 seq 373662154754 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154755 -lt 373662154754 2024-02-05T05:05:59.711 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416392 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416392 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416392 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416392' 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: waiting osd.2 seq 339302416392 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416392 -lt 339302416392 2024-02-05T05:06:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089346 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089346 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089346' 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: waiting osd.3 seq 382252089346 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089346 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:00.955 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:06:00.956 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.c65: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416391 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416391 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416391 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416391' 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: waiting osd.2 seq 339302416391 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:06:01.020 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416392 -lt 339302416391 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089347 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089347 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089347 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089347' 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: waiting osd.3 seq 382252089347 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089347 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:01.021 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:06:01.022 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:02.384 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:02.384 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:02.384 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:06:02.384 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:06:02.384 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:02.385 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-02-05T05:06:02.386 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-02-05T05:06:02.470 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:02.471 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:06:02.472 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:06:02.473 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:06:02.473 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:06:02.473 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:06:02.473 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-02-05T05:06:02.473 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-02-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-02-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:171467: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:06:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:06:03.251 INFO:tasks.workunit.client.0.smithi188.stderr:client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-02-05T05:06:03.251 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:06:03.251 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:06:03.251 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:06:03.251 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:03.251 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:03.251 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:171465: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:06:03.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171464 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:06:03.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:06:03.254 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:06:03.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:06:03.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:05:31.786501+0000 2024-02-05T05:06:03.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:06:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:06:04.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:05:31.786501+0000 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:05:31.786501+0000 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:06:04.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:06:04.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:31.786501+0000 '>' 2024-02-05T05:05:31.786501+0000 2024-02-05T05:06:04.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:06:05.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:06:05.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:06:05.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:06:05.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:06:05.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:06:05.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:06:05.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:06:05.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:05:31.786501+0000 '>' 2024-02-05T05:05:31.786501+0000 2024-02-05T05:06:05.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:06:06.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:06:06.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:06:06.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:06:06.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:06:06.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:06:06.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:06:06.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:06:06.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:06:04.509632+0000 '>' 2024-02-05T05:05:31.786501+0000 2024-02-05T05:06:06.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:06:06.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:06:06.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:06:06.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174509"' 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174509' 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174511"' 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174511' 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 174509 174511' 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:06:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:06:06.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174509 2024-02-05T05:06:06.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174512: /' 2024-02-05T05:06:06.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:06:06.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174514: /' 2024-02-05T05:06:09.997 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:06:09.997 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:06:09.997 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:06:09.997 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:06:09.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: _ 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: hinfo_key 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: snapset 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:06:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:06:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:06:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:06:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:06:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:06:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:06:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:06:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:06:10.001 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:06:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:174512: start osd.3 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:06:10.355 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:06:10.356 INFO:tasks.workunit.client.0.smithi188.stderr:174514: _ 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: hinfo_key 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: snapset 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:06:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:10.358 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:06:10.359 INFO:tasks.workunit.client.0.smithi188.stderr:174514: start osd.1 2024-02-05T05:06:14.134 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:06:14.134 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:06:14.134 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:06:14.134 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:06:14.134 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:06:14.134 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:06:14.134 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 2024-02-05T05:06:10.027+0000 7f3534e0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 2024-02-05T05:06:10.030+0000 7f3534e0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 2024-02-05T05:06:10.031+0000 7f3534e0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 0 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.135 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 2024-02-05T05:06:10.550+0000 7f3534e0f4c0 -1 Falling back to public interface 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 1 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:14.136 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 2 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 2024-02-05T05:06:13.069+0000 7f3534e0f4c0 -1 osd.3 91 log_to_monitors true 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 3 2024-02-05T05:06:14.137 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.920 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-02-05T05:06:14.920 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:06:14.920 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 2024-02-05T05:06:10.381+0000 7fb4cdc164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 2024-02-05T05:06:10.404+0000 7fb4cdc164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 2024-02-05T05:06:10.406+0000 7fb4cdc164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:06:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 0 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 2024-02-05T05:06:10.927+0000 7fb4cdc164c0 -1 Falling back to public interface 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 1 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:06:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 2 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 2024-02-05T05:06:13.647+0000 7fb4cdc164c0 -1 osd.1 91 log_to_monitors true 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 3 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:06:15.438 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:06:15.438 INFO:tasks.workunit.client.0.smithi188.stderr:174512: osd.3 up in weight 1 up_from 96 up_thru 89 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6818/2793668942,v1:127.0.0.1:6819/2793668942] [v2:127.0.0.1:6820/2793668942,v1:127.0.0.1:6821/2793668942] exists,up 79318c2b-6f97-4afb-b90f-b6754bde4320 2024-02-05T05:06:15.438 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:06:15.439 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 1 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 2 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: 3' 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836545 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836545 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545' 2024-02-05T05:06:15.440 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:06:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:06:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-02-05T05:06:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-02-05T05:06:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-425201762306' 2024-02-05T05:06:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd ieph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 4 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: osd.1 up in weight 1 up_from 99 up_thru 99 down_at 92 last_clean_interval [87,91) [v2:127.0.0.1:6826/77365308,v1:127.0.0.1:6827/77365308] [v2:127.0.0.1:6828/77365308,v1:127.0.0.1:6829/77365308] exists,up b543dab3-3e88-41a2-a182-082434b3e8c1 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:06:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:174514: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 1 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 2 2024-02-05T05:06:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:174514: 3' 2024-02-05T05:06:16.942 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:06:16.942 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:06:16.942 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:06:16.942 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-02-05T05:06:16.942 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-02-05T05:06:16.942 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-02-05T05:06:16.942 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:06:17.997 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hn $ids 2024-02-05T05:06:17.997 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416396 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416396 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-425201762306 2-339302416396' 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-425201762306 2-339302416396 3-412316860418' 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836545 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836545 2024-02-05T05:06:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836545 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: waiting osd.0 seq 21474836545 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836545 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836545 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:06:18.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-02-05T05:06:18.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:18.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-02-05T05:06:18.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-02-05T05:06:18.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: waiting osd.1 seq 425201762306 2024-02-05T05:06:18.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:06:18.000 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174512elpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762307 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762307 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-425201762307' 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416398 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416398 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-425201762307 2-339302416398' 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:06:18.811 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-425201762307 2-339302416398 3-412316860419' 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-02-05T05:06:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:174514: waiting osd.0 seq 21474836546 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836546 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836546 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762307 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762307 2024-02-05T05:06:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416396 2024-02-05T05:06:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:06:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416396 2024-02-05T05:06:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416396 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416396' 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: waiting osd.2 seq 339302416396 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416397 -lt 339302416396 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-412316860418 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-412316860418 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 412316860418' 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: waiting osd.3 seq 412316860418 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:06:19.201 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860418 -lt 412316860418 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:19.202 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:06:19.566 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:19.566 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:19.566 INFO:tasks.workunit.client.0.smithi188.stderr:174512: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:19.567 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:06:19.567 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:06:19.567 INFO:tasks.workunit.client.0.smithi188.stderr:174512: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:06:19.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:06:19.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:06:19.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174511 2024-02-05T05:06:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762307 2024-02-05T05:06:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762307' 2024-02-05T05:06:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:174514: waiting osd.1 seq 425201762307 2024-02-05T05:06:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:06:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762307 -lt 425201762307 2024-02-05T05:06:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-339302416398 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-339302416398 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416398 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 339302416398' 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: waiting osd.2 seq 339302416398 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416398 -lt 339302416398 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-412316860419 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:06:20.083 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-412316860419 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 412316860419' 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: waiting osd.3 seq 412316860419 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860419 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:06:20.084 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:06:20.689 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:174514: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:06:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:06:20.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:06:20.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:06:20.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:06:20.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:06:20.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:06:20.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:06:20.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:06:20.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:06:20.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:06:20.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:06:20.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:06:20.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:06:20.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:06:20.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:06:20.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:06:20.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:06:20.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:06:20.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:06:20.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:06:20.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:06:20.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:20.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:20.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:06:20.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:06:20.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:06:20.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:06:20.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:06:20.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:06:20.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:06:20.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:06:20.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:06:20.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:06:20.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:06:20.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:06:20.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:06:20.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:06:20.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:06:20.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:06:20.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:06:20.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:06:20.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:06:20.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:06:20.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:06:20.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:06:20.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:20.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:06:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:06:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:06:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:06:20.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:06:20.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:20.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:06:20.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:06:20.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:06:20.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:21.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:06:21.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:06:21.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:06:21.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:06:21.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:06:21.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:06:21.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:06:21.466 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:06:21.467 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:06:21.467 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:06:21.467 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:06:21.468 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:21.468 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:21.468 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:06:21.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:06:21.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:21.548 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:21.549 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:06:21.549 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:06:21.549 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:06:21.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-02-05T05:06:21.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:06:21.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:06:21.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:06:21.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:06:21.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:06:21.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:06:22.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:06:22.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:06:22.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-02-05T05:06:22.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:06:22.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-02-05T05:06:22.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:06:22.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:06:22.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:06:22.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:06:22.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:06:22.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:06:22.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:06:22.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:06:22.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:06:22.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3066c3f9-b402-478e-bf62-bd432b4aff34 2024-02-05T05:06:22.477 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 3066c3f9-b402-478e-bf62-bd432b4aff34 2024-02-05T05:06:22.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3066c3f9-b402-478e-bf62-bd432b4aff34' 2024-02-05T05:06:22.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:06:22.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDObMBlzKo3HRAAs/URQFpDlVo1HgUWnVbZ8Q== 2024-02-05T05:06:22.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDObMBlzKo3HRAAs/URQFpDlVo1HgUWnVbZ8Q=="}' 2024-02-05T05:06:22.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3066c3f9-b402-478e-bf62-bd432b4aff34 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:06:22.699 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:06:22.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:06:22.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDObMBlzKo3HRAAs/URQFpDlVo1HgUWnVbZ8Q== --osd-uuid 3066c3f9-b402-478e-bf62-bd432b4aff34 2024-02-05T05:06:22.737 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:22.736+0000 7fb8f4e0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:22.750 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:22.748+0000 7fb8f4e0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:22.750 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:22.749+0000 7fb8f4e0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:22.751 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:22.749+0000 7fb8f4e0d4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:06:22.751 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:22.749+0000 7fb8f4e0d4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:06:22.751 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:22.749+0000 7fb8f4e0d4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:06:22.842 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:22.840+0000 7fb8f4e0d4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:06:28.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:06:28.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:06:28.342 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:06:28.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:06:28.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:06:28.690 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:06:28.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:06:28.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:06:28.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:06:28.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:06:28.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:06:28.716 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:28.714+0000 7fe6c53074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:28.718 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:28.716+0000 7fe6c53074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:28.719 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:28.717+0000 7fe6c53074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:28.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:06:28.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:06:28.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:06:28.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:06:28.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:06:28.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:28.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:06:28.959 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:06:28.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:28.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:29.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:29.240 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:29.238+0000 7fe6c53074c0 -1 Falling back to public interface 2024-02-05T05:06:30.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:30.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:30.225 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:06:30.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:06:30.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:30.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:30.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:31.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:31.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:31.495 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:06:31.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:06:31.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:31.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:31.574 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:31.572+0000 7fe6c53074c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:06:31.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:32.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:32.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:32.766 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:06:32.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:06:32.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:32.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:33.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:34.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:34.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:34.043 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:06:34.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:06:34.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:34.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:34.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:35.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:35.327 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:06:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:06:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:35.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:36.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:36.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:36.608 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:06:36.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:06:36.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:36.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:37.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:37.885 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:06:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:06:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:37.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:38.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:39.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:39.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:39.163 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:06:39.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:06:39.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:39.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:39.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:40.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:40.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:40.523 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:06:40.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:06:40.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:40.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:40.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:41.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:41.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:41.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:06:41.803 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:06:41.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:41.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:42.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:43.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:43.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:43.102 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:06:43.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:06:43.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:43.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:43.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:44.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:44.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:44.924 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:06:44.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:06:44.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:44.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:45.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:46.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:46.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:46.431 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:06:46.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:06:46.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:46.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:47.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:48.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:48.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:48.103 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:06:48.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:06:48.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:48.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:48.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:49.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:49.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:49.606 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:06:49.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:06:49.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:49.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:50.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:51.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:51.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:51.232 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:06:51.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:06:51.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:51.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:51.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:52.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:52.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:52.609 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:06:52.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:06:52.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:52.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:53.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:06:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:06:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:06:54.088 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:06:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:06:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:06:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:06:54.356 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4291599376,v1:127.0.0.1:6803/4291599376] [v2:127.0.0.1:6804/4291599376,v1:127.0.0.1:6805/4291599376] exists,up 3066c3f9-b402-478e-bf62-bd432b4aff34 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:06:54.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:06:54.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:06:54.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:06:54.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:06:54.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 73d76fcf-c4a3-454e-a5f0-53de684d4cc0' 2024-02-05T05:06:54.365 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:06:54.365 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:06:54.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDubMBlksxrFhAAf8tNRB0raA1PwGCgru8SQA== 2024-02-05T05:06:54.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDubMBlksxrFhAAf8tNRB0raA1PwGCgru8SQA=="}' 2024-02-05T05:06:54.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:06:54.740 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:06:54.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:06:54.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDubMBlksxrFhAAf8tNRB0raA1PwGCgru8SQA== --osd-uuid 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:06:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:54.777+0000 7f63459a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:54.816 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:54.814+0000 7f63459a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:54.816 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:54.815+0000 7f63459a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:06:54.817 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:54.815+0000 7f63459a94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:06:54.817 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:54.815+0000 7f63459a94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:06:54.817 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:54.815+0000 7f63459a94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:06:54.891 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:06:54.889+0000 7f63459a94c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:07:00.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:07:00.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:07:00.305 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:07:00.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:07:00.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:07:00.683 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:07:00.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:07:00.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:07:00.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:07:00.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:07:00.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:07:00.709 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:00.707+0000 7fe03b79b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:00.711 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:00.709+0000 7fe03b79b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:00.712 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:00.710+0000 7fe03b79b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:00.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:07:00.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:07:00.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:07:00.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:07:00.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:07:00.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:00.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:07:00.992 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:07:00.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:00.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:01.242+0000 7fe03b79b4c0 -1 Falling back to public interface 2024-02-05T05:07:01.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:02.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:02.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:02.261 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:07:02.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:07:02.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:02.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:02.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:03.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:03.569 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:07:03.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:03.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:07:03.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:03.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:03.589 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:03.587+0000 7fe03b79b4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:07:03.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:04.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:04.916 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:07:04.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:04.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:07:04.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:04.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:05.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:06.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:06.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:06.205 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:07:06.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:07:06.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:06.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:06.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:07.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:07.596 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:07:07.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:07.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:07:07.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:07.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:07.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:08.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:08.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:08.874 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:07:08.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:07:08.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:08.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:09.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:10.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:10.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:10.183 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:07:10.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:07:10.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:10.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:10.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:11.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:11.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:11.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:07:11.539 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:07:11.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:11.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:11.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:12.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:12.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:12.815 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:07:12.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:07:12.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:12.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:13.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:14.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:14.093 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:07:14.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:14.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:07:14.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:14.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:14.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:07:15.441 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:07:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:15.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:15.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:16.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:16.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:16.997 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:07:16.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:07:16.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:16.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:17.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:18.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:18.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:18.353 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:07:18.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:07:18.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:18.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:18.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:19.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:19.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:19.831 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:07:19.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:07:19.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:19.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:20.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:21.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:21.387 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:07:21.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:21.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:07:21.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:21.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:21.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:22.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:22.686 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:07:22.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:22.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:07:22.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:22.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:23.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:24.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:24.253 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:07:24.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:24.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:07:24.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:24.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:24.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:25.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:25.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:25.931 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:07:25.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:07:25.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:25.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:26.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:27.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:27.238 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:07:27.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:27.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:07:27.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:27.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:07:27.513 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/918848163,v1:127.0.0.1:6811/918848163] [v2:127.0.0.1:6812/918848163,v1:127.0.0.1:6813/918848163] exists,up 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:07:27.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:07:27.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:07:27.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:07:27.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:07:27.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-02-05T05:07:27.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:07:27.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:07:27.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:07:27.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:07:27.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:07:27.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:07:27.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:07:27.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:07:27.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:07:27.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:07:27.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:07:27.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:07:27.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:07:27.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=60c7027b-262c-4328-bdf9-a5c87f3e89d2 2024-02-05T05:07:27.522 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 60c7027b-262c-4328-bdf9-a5c87f3e89d2 2024-02-05T05:07:27.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 60c7027b-262c-4328-bdf9-a5c87f3e89d2' 2024-02-05T05:07:27.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:07:27.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPbcBlxJi4HxAAMkk07IVOGDUBicGBPpTcxQ== 2024-02-05T05:07:27.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPbcBlxJi4HxAAMkk07IVOGDUBicGBPpTcxQ=="}' 2024-02-05T05:07:27.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 60c7027b-262c-4328-bdf9-a5c87f3e89d2 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:07:27.887 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:07:27.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:07:27.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPbcBlxJi4HxAAMkk07IVOGDUBicGBPpTcxQ== --osd-uuid 60c7027b-262c-4328-bdf9-a5c87f3e89d2 2024-02-05T05:07:27.928 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:27.925+0000 7f98e2cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:27.972 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:27.970+0000 7f98e2cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:27.973 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:27.971+0000 7f98e2cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:27.974 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:27.972+0000 7f98e2cb64c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:07:27.974 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:27.972+0000 7f98e2cb64c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:07:27.974 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:27.972+0000 7f98e2cb64c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:07:28.047 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:28.045+0000 7f98e2cb64c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:07:34.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:07:34.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:07:34.029 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:07:34.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:07:34.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:07:34.375 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:07:34.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:07:34.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:07:34.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:07:34.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:07:34.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:07:34.401 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:34.399+0000 7f96d90cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:34.403 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:34.401+0000 7f96d90cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:34.404 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:34.402+0000 7f96d90cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:07:34.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:07:34.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:07:34.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:07:34.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:07:34.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:07:34.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:34.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:07:34.650 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:07:34.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:34.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:34.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:34.934 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:34.932+0000 7f96d90cf4c0 -1 Falling back to public interface 2024-02-05T05:07:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:35.918 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:07:35.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:07:35.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:35.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:36.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:37.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:37.193 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:07:37.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:37.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:07:37.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:37.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:37.272 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:07:37.270+0000 7f96d90cf4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:07:37.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:38.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:38.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:38.473 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:07:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:07:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:38.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:39.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:39.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:39.848 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:07:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:07:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:40.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:41.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:41.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:41.219 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:07:41.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:07:41.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:41.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:41.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:42.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:42.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:42.500 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:07:42.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:07:42.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:42.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:42.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:43.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:43.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:43.813 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:07:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:07:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:44.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:45.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:45.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:45.092 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:07:45.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:07:45.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:45.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:45.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:46.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:46.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:46.368 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:07:46.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:07:46.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:46.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:46.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:47.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:47.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:47.677 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:07:47.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:07:47.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:47.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:47.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:48.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:48.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:48.954 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:07:48.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:07:48.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:48.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:49.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:50.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:50.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:50.232 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:07:50.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:07:50.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:50.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:50.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:51.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:51.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:51.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:07:51.785 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:07:51.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:51.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:52.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:53.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:53.071 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:07:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:07:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:53.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:54.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:54.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:54.765 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:07:54.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:07:54.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:54.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:55.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:56.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:56.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:56.480 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:07:56.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:07:56.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:56.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:57.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:07:58.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:07:58.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:07:58.546 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:07:58.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:07:58.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:07:58.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:07:59.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:00.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:00.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:00.332 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:08:00.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:08:00.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:00.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:08:00.675 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3013071197,v1:127.0.0.1:6819/3013071197] [v2:127.0.0.1:6820/3013071197,v1:127.0.0.1:6821/3013071197] exists,up 60c7027b-262c-4328-bdf9-a5c87f3e89d2 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:08:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:08:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:08:00.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:08:00.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:08:00.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:08:00.682 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:08:00.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47' 2024-02-05T05:08:00.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:08:00.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwbcBlVelgKRAAf+kI4NcFPxU70Uwzqjrh/A== 2024-02-05T05:08:00.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwbcBlVelgKRAAf+kI4NcFPxU70Uwzqjrh/A=="}' 2024-02-05T05:08:00.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 -i td/osd-scrub-repair/3/new.json 2024-02-05T05:08:01.025 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:08:01.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:08:01.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwbcBlVelgKRAAf+kI4NcFPxU70Uwzqjrh/A== --osd-uuid 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:08:01.065 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:01.063+0000 7fb1c04254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:01.102 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:01.100+0000 7fb1c04254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:01.103 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:01.101+0000 7fb1c04254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:01.103 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:01.101+0000 7fb1c04254c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:08:01.103 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:01.101+0000 7fb1c04254c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:08:01.103 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:01.101+0000 7fb1c04254c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:08:01.168 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:01.166+0000 7fb1c04254c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:08:07.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:08:07.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:08:07.479 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T05:08:07.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:08:07.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:08:07.824 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T05:08:07.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:08:07.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:08:07.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:08:07.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:08:07.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:08:07.850 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:07.848+0000 7f1b0ad6e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:07.852 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:07.850+0000 7f1b0ad6e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:07.853 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:07.851+0000 7f1b0ad6e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:08.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:08:08.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:08:08.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:08:08.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:08:08.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:08:08.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:08.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:08:08.093 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:08:08.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:08.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:08.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:08.376 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:08.374+0000 7f1b0ad6e4c0 -1 Falling back to public interface 2024-02-05T05:08:09.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:09.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:09.357 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:08:09.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:08:09.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:09.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:09.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:10.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:10.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:10.637 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:08:10.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:08:10.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:10.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:10.853 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:10.851+0000 7f1b0ad6e4c0 -1 osd.3 0 log_to_monitors true 2024-02-05T05:08:10.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:11.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:11.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:11.946 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:08:11.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:08:11.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:11.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:12.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:13.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:13.220 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:08:13.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:13.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:08:13.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:13.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:13.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:14.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:14.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:14.497 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:08:14.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:08:14.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:14.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:14.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:15.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:15.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:15.862 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:08:15.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:08:15.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:15.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:16.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:17.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:17.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:17.142 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:08:17.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:08:17.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:17.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:17.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:18.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:18.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:18.426 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:08:18.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:08:18.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:18.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:18.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:19.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:19.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:19.880 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:08:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:08:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:20.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:21.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:21.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:21.159 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:08:21.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:08:21.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:21.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:21.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:22.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:22.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:22.461 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:08:22.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:08:22.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:22.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:22.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:23.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:23.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:23.838 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:08:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:08:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:24.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:25.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:25.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:25.696 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:08:25.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:08:25.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:25.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:26.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:27.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:27.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:27.240 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:08:27.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:08:27.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:27.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:27.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:28.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:28.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:28.862 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:08:28.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:08:28.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:28.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:29.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:30.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:30.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:30.505 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:08:30.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:08:30.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:30.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:31.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:32.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:32.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:32.267 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:08:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:08:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:33.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:34.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:34.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:34.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:08:34.077 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:08:34.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:34.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:34.355 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/901302385,v1:127.0.0.1:6827/901302385] [v2:127.0.0.1:6828/901302385,v1:127.0.0.1:6829/901302385] exists,up 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:08:34.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:08:34.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:08:34.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:08:34.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-02-05T05:08:34.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:08:34.615 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:08:34.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:08:34.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:08:34.988 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:08:34.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:08:36.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:08:37.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-02-05T05:08:37.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:08:37.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:08:37.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:08:37.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:08:37.813 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:08:37.813 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:08:37.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:08:37.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:08:37.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:08:37.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:08:37.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:08:37.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:08:37.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:08:37.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:08:37.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:08:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:08:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:08:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:08:38.177 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:08:38.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:08:38.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:38.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:08:38.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-05T05:08:38.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-05T05:08:38.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-05T05:08:38.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:38.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:08:38.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-02-05T05:08:38.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-02-05T05:08:38.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672982' 2024-02-05T05:08:38.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:38.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:08:38.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-02-05T05:08:38.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-02-05T05:08:38.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509454' 2024-02-05T05:08:38.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:38.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:08:38.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345925 2024-02-05T05:08:38.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345925 2024-02-05T05:08:38.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672982 2-64424509454 3-85899345925' 2024-02-05T05:08:38.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:38.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-05T05:08:38.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:38.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:08:38.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-05T05:08:38.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:38.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-05T05:08:38.589 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836505 2024-02-05T05:08:38.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-05T05:08:38.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:08:38.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-02-05T05:08:38.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:38.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-02-05T05:08:38.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:38.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:08:38.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-02-05T05:08:38.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:38.865 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672982 2024-02-05T05:08:38.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-02-05T05:08:38.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-02-05T05:08:38.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:08:39.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672982 2024-02-05T05:08:39.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:08:40.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:08:40.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:08:40.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672982 2024-02-05T05:08:40.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:08:41.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:08:41.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:08:41.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-02-05T05:08:41.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:41.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-02-05T05:08:41.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:41.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:08:41.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-02-05T05:08:41.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:41.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-02-05T05:08:41.686 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509454 2024-02-05T05:08:41.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-02-05T05:08:41.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:08:41.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-02-05T05:08:41.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:41.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345925 2024-02-05T05:08:41.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:41.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:08:41.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345925 2024-02-05T05:08:41.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:41.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345925 2024-02-05T05:08:41.957 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345925 2024-02-05T05:08:41.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-02-05T05:08:41.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:08:42.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345926 -lt 85899345925 2024-02-05T05:08:42.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:08:42.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:08:42.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:08:42.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:08:42.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:08:42.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:08:42.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:08:42.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:08:42.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:08:42.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:08:42.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:08:42.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:08:42.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:08:42.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:08:42.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:08:43.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T05:08:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:08:43.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:08:44.119 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:08:44.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:08:45.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:08:45.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:08:45.592 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T05:08:45.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:08:45.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:08:45.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:08:45.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:08:45.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:08:45.605 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:08:45.605 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:08:45.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:08:45.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:08:45.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:08:45.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:08:45.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:08:45.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:08:45.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:08:45.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:08:45.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:08:46.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:08:46.025 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:08:46.025 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:08:46.025 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:08:46.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:08:46.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:46.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:08:46.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-05T05:08:46.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-05T05:08:46.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-05T05:08:46.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:46.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:08:46.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-02-05T05:08:46.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-02-05T05:08:46.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672984' 2024-02-05T05:08:46.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:46.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:08:46.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509457 2024-02-05T05:08:46.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509457 2024-02-05T05:08:46.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672984 2-64424509457' 2024-02-05T05:08:46.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:46.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:08:46.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-02-05T05:08:46.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-02-05T05:08:46.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508 1-42949672984 2-64424509457 3-85899345928' 2024-02-05T05:08:46.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:46.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-05T05:08:46.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:46.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:08:46.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-05T05:08:46.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:46.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-05T05:08:46.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-05T05:08:46.542 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836508 2024-02-05T05:08:46.542 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:08:46.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-05T05:08:46.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:46.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-02-05T05:08:46.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:46.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:08:46.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-02-05T05:08:46.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:46.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-02-05T05:08:46.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-02-05T05:08:46.812 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672984 2024-02-05T05:08:46.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:08:47.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-02-05T05:08:47.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:47.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509457 2024-02-05T05:08:47.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:47.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:08:47.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509457 2024-02-05T05:08:47.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:47.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509457 2024-02-05T05:08:47.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-02-05T05:08:47.096 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509457 2024-02-05T05:08:47.096 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:08:47.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509457 2024-02-05T05:08:47.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:08:48.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:08:48.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:08:48.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509457 2024-02-05T05:08:48.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345928 2024-02-05T05:08:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:48.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:08:48.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345928 2024-02-05T05:08:48.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-02-05T05:08:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-02-05T05:08:48.663 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345928 2024-02-05T05:08:48.663 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:08:48.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345928 2024-02-05T05:08:48.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:08:48.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:08:48.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:08:49.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:08:49.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:08:49.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:08:49.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:08:49.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:08:49.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:08:49.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:08:49.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:08:49.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:08:49.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:08:49.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:08:49.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:08:49.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:08:49.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:08:49.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:08:49.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:08:50.393 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:08:50.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:08:50.762 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:08:50.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:08:50.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:08:50.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:08:51.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:08:51.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:08:51.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:08:51.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:08:51.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:08:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:08:51.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:08:51.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:08:51.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:08:51.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:08:51.334 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:08:51.334 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:08:51.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:08:51.602 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:08:51.602 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:08:51.602 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:08:51.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:08:51.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:08:51.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:08:51.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:08:51.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:08:51.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:08:51.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:08:51.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:08:51.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:08:51.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:08:51.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:08:51.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:08:51.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:08:53.547 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:08:54.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:08:54.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:08:54.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:08:54.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:08:54.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:08:54.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:08:54.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:08:54.069 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:08:54.070 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:08:54.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:08:54.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:08:54.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:08:54.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:08:54.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:08:54.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:08:54.098 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:54.096+0000 7f8546f8b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:54.100 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:54.098+0000 7f8546f8b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:54.100+0000 7f8546f8b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:54.617 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:54.615+0000 7f8546f8b4c0 -1 Falling back to public interface 2024-02-05T05:08:54.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:08:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:08:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:55.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:56.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:56.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:56.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:08:56.927 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:08:56.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:56.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:57.062 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:08:57.060+0000 7f8546f8b4c0 -1 osd.3 36 log_to_monitors true 2024-02-05T05:08:57.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:08:58.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:08:58.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:08:58.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:08:58.267 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:08:58.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:08:58.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:08:58.682 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/3922802458,v1:127.0.0.1:6827/3922802458] [v2:127.0.0.1:6828/3922802458,v1:127.0.0.1:6829/3922802458] exists,up 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:08:58.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:08:58.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:08:58.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:08:58.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:08:58.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:08:58.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:08:58.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:08:58.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:08:58.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:08:58.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:08:59.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:08:59.060 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:08:59.060 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:08:59.060 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:08:59.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:08:59.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:59.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:08:59.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836512 2024-02-05T05:08:59.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836512 2024-02-05T05:08:59.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512' 2024-02-05T05:08:59.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:59.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:08:59.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-02-05T05:08:59.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-02-05T05:08:59.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672988' 2024-02-05T05:08:59.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:59.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:08:59.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-02-05T05:08:59.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-02-05T05:08:59.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672988 2-64424509460' 2024-02-05T05:08:59.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:08:59.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:08:59.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-05T05:08:59.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-05T05:08:59.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836512 1-42949672988 2-64424509460 3-171798691842' 2024-02-05T05:08:59.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:08:59.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836512 2024-02-05T05:08:59.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:08:59.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:08:59.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836512 2024-02-05T05:08:59.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:08:59.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836512 2024-02-05T05:08:59.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-02-05T05:08:59.502 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836512 2024-02-05T05:08:59.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:08:59.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836512 2024-02-05T05:08:59.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:09:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:09:00.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:01.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836512 2024-02-05T05:09:01.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:01.043 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-02-05T05:09:01.043 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:01.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:09:01.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-02-05T05:09:01.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:01.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-02-05T05:09:01.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-02-05T05:09:01.046 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672988 2024-02-05T05:09:01.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:01.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-02-05T05:09:01.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:01.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-02-05T05:09:01.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:01.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:09:01.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-02-05T05:09:01.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:01.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-02-05T05:09:01.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-02-05T05:09:01.320 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509460 2024-02-05T05:09:01.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:09:01.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-02-05T05:09:01.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:01.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-02-05T05:09:01.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:01.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:09:01.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-02-05T05:09:01.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:01.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-05T05:09:01.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-02-05T05:09:01.593 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 171798691842 2024-02-05T05:09:01.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:09:01.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-02-05T05:09:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:09:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:02.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:09:02.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:09:02.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:09:02.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:09:02.198 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:09:02.198 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:09:02.198 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:09:02.198 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:09:02.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:09:02.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:09:02.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:02.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:02.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:09:02.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:09:02.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:09:02.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:09:02.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:09:02.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:09:02.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:09:02.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:09:03.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:09:03.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:09:03.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:09:03.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:09:03.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:03.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:03.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:03.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:03.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:03.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:09:03.681 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:09:03.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:03.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:09:03.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:03.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:09:03.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:09:03.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:03.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:03.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:03.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:03.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:03.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:03.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:08:44.016624+0000 '>' 2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:03.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:09:04.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:09:04.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:04.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:04.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:04.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:04.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:04.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:05.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:08:44.016624+0000 '>' 2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:05.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:09:06.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:09:06.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:06.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:06.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:06.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:06.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:06.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:06.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:08:44.016624+0000 '>' 2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:06.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:09:07.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:09:07.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:07.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:07.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:07.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:07.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:07.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:07.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:08:44.016624+0000 '>' 2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:07.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:09:08.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:09:08.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:08.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:08.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:08.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:08.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:08.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:04.124550+0000 '>' 2024-02-05T05:08:44.016624+0000 2024-02-05T05:09:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:09:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:09:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:09:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:09:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:09:09.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:09:09.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:09:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:09:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:09:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:09:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:09:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:09:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:09:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:09:11.096 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:09:11.096 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:09:11.096 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:09:11.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:09:11.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:09:11.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:09:11.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:09:11.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:09:11.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:09:11.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:09:11.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:09:11.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:09:11.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:09:11.393 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:11.391+0000 7f368eebe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:11.393+0000 7f368eebe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:11.394+0000 7f368eebe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:11.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:09:11.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:09:11.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:09:11.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:09:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:09:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:09:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:09:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:11.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:11.920 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:11.918+0000 7f368eebe4c0 -1 Falling back to public interface 2024-02-05T05:09:12.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:12.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:12.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:09:12.909 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:09:12.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:12.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:13.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:14.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:14.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:14.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:09:14.179 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:09:14.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:14.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:14.215 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:14.213+0000 7f368eebe4c0 -1 osd.3 42 log_to_monitors true 2024-02-05T05:09:14.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:15.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:15.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:15.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:09:15.496 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:09:15.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:15.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:15.774 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 47 up_thru 47 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/2581901356,v1:127.0.0.1:6827/2581901356] [v2:127.0.0.1:6828/2581901356,v1:127.0.0.1:6829/2581901356] exists,up 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:09:15.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:15.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:09:15.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:09:15.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:09:15.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:09:15.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:09:15.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:09:15.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:09:15.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:09:16.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:09:16.227 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:09:16.227 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:09:16.227 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:09:16.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:09:16.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:16.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:09:16.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-02-05T05:09:16.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-02-05T05:09:16.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-02-05T05:09:16.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:16.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:09:16.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672992 2024-02-05T05:09:16.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672992 2024-02-05T05:09:16.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672992' 2024-02-05T05:09:16.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:16.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:09:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-02-05T05:09:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-02-05T05:09:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672992 2-64424509465' 2024-02-05T05:09:16.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:16.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:09:16.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462914 2024-02-05T05:09:16.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462914 2024-02-05T05:09:16.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672992 2-64424509465 3-201863462914' 2024-02-05T05:09:16.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:16.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-02-05T05:09:16.640 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:16.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:09:16.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-02-05T05:09:16.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:16.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-02-05T05:09:16.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-02-05T05:09:16.642 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836516 2024-02-05T05:09:16.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:16.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-02-05T05:09:16.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:16.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672992 2024-02-05T05:09:16.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:16.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:09:16.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672992 2024-02-05T05:09:16.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:16.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672992 2024-02-05T05:09:16.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-02-05T05:09:16.910 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672992 2024-02-05T05:09:16.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:17.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672992 2024-02-05T05:09:17.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:09:18.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:09:18.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:18.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672992 2024-02-05T05:09:18.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:09:19.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:09:19.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:19.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672993 -lt 42949672992 2024-02-05T05:09:19.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:19.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-02-05T05:09:19.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:19.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:09:19.746 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-02-05T05:09:19.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:19.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-02-05T05:09:19.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-02-05T05:09:19.748 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509465 2024-02-05T05:09:19.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:09:20.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-02-05T05:09:20.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:20.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462914 2024-02-05T05:09:20.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:20.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:09:20.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462914 2024-02-05T05:09:20.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:20.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462914 2024-02-05T05:09:20.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462914' 2024-02-05T05:09:20.018 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 201863462914 2024-02-05T05:09:20.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:09:20.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462914 -lt 201863462914 2024-02-05T05:09:20.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:09:20.284 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:20.284 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:20.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:09:20.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:09:20.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:09:20.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:09:20.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:09:20.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:09:20.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:09:20.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:09:20.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:09:20.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:09:20.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:20.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:21.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:09:21.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:09:21.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:09:21.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:09:21.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:09:21.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-02-05T05:09:21.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:09:21.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:09:21.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:09:21.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:09:21.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:09:21.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:09:21.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:09:23.383 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:09:23.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:09:23.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:09:23.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:09:23.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:09:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:09:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:09:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:09:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:09:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:09:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:09:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:09:23.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:09:23.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:09:23.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:09:23.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:09:23.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:09:23.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:09:23.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:09:23.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:09:23.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:09:23.933 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:23.931+0000 7f3c8feb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:23.945 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:23.933+0000 7f3c8feb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:23.946 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:23.934+0000 7f3c8feb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:09:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:24.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:24.451 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:24.449+0000 7f3c8feb44c0 -1 Falling back to public interface 2024-02-05T05:09:24.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:25.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:25.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:25.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:09:25.485 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:09:25.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:25.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:25.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:26.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:26.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:26.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:09:26.761 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:09:26.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:26.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:27.019 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:27.017+0000 7f3c8feb44c0 -1 osd.1 49 log_to_monitors true 2024-02-05T05:09:27.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:28.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:28.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:28.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:09:28.034 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:09:28.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:28.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 54 up_thru 54 down_at 50 last_clean_interval [10,49) [v2:127.0.0.1:6810/3939561095,v1:127.0.0.1:6811/3939561095] [v2:127.0.0.1:6812/3939561095,v1:127.0.0.1:6813/3939561095] exists,up 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:09:28.315 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:09:28.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:09:28.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:09:28.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:09:28.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:09:28.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:09:28.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:09:28.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:09:28.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:09:28.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:09:28.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:09:28.719 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:09:28.719 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:09:28.719 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:09:28.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:09:28.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:28.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:09:28.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-02-05T05:09:28.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-02-05T05:09:28.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-02-05T05:09:28.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:28.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:09:29.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233986 2024-02-05T05:09:29.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233986 2024-02-05T05:09:29.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-231928233986' 2024-02-05T05:09:29.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:29.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:09:29.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-02-05T05:09:29.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-02-05T05:09:29.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-231928233986 2-64424509468' 2024-02-05T05:09:29.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:29.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:09:29.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462917 2024-02-05T05:09:29.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462917 2024-02-05T05:09:29.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-231928233986 2-64424509468 3-201863462917' 2024-02-05T05:09:29.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:29.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-02-05T05:09:29.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:29.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:09:29.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-02-05T05:09:29.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:29.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-02-05T05:09:29.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-02-05T05:09:29.351 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836519 2024-02-05T05:09:29.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:29.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836519 2024-02-05T05:09:29.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:09:30.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:09:30.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:30.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836519 2024-02-05T05:09:30.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:30.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233986 2024-02-05T05:09:30.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:30.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:09:30.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233986 2024-02-05T05:09:30.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:30.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233986 2024-02-05T05:09:30.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233986' 2024-02-05T05:09:30.895 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 231928233986 2024-02-05T05:09:30.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:31.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233986 -lt 231928233986 2024-02-05T05:09:31.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:31.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-02-05T05:09:31.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:31.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:09:31.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-02-05T05:09:31.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:31.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-02-05T05:09:31.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-02-05T05:09:31.167 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509468 2024-02-05T05:09:31.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:09:31.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-02-05T05:09:31.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:31.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-201863462917 2024-02-05T05:09:31.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:31.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:09:31.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-201863462917 2024-02-05T05:09:31.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:31.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462917 2024-02-05T05:09:31.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 201863462917' 2024-02-05T05:09:31.437 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 201863462917 2024-02-05T05:09:31.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:09:31.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462918 -lt 201863462917 2024-02-05T05:09:31.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:09:31.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:31.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:09:32.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:09:32.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:09:32.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:09:32.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:32.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:32.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:09:32.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:09:32.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:09:32.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:09:32.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:09:32.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:09:32.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:09:32.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:32.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:33.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:09:04.124550+0000 2024-02-05T05:09:33.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:09:33.419 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:09:33.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:09:04.124550+0000 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:09:04.124550+0000 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:33.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:33.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:04.124550+0000 '>' 2024-02-05T05:09:04.124550+0000 2024-02-05T05:09:33.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:09:34.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:09:34.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:34.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:34.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:34.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:34.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:34.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:34.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:04.124550+0000 '>' 2024-02-05T05:09:04.124550+0000 2024-02-05T05:09:34.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:09:35.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:09:35.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:35.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:35.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:35.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:35.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:35.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:36.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:04.124550+0000 '>' 2024-02-05T05:09:04.124550+0000 2024-02-05T05:09:36.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:09:37.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:09:37.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:09:37.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:09:37.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:09:37.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:09:37.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:09:37.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:34.182940+0000 '>' 2024-02-05T05:09:04.124550+0000 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:09:37.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:09:37.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:09:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:09:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:09:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:09:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:09:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:09:37.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:09:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:09:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:09:39.200 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:09:39.200 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:09:39.201 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:09:39.201 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:09:39.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:09:39.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:09:39.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:09:39.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:09:39.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:09:39.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:09:39.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:09:39.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:09:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:39.496+0000 7efc40c634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:39.498+0000 7efc40c634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:39.499+0000 7efc40c634c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:39.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:09:39.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:09:39.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:09:39.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:09:39.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:09:39.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:39.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:09:39.749 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:09:39.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:39.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:40.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:40.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:40.026+0000 7efc40c634c0 -1 Falling back to public interface 2024-02-05T05:09:41.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:41.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:41.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:09:41.015 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:09:41.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:41.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:41.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:42.187 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:09:42.185+0000 7efc40c634c0 -1 osd.3 56 log_to_monitors true 2024-02-05T05:09:42.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:42.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:42.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:09:42.288 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:09:42.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:42.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:42.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:43.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:43.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:43.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:09:43.584 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:09:43.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:43.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:09:43.947 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [47,56) [v2:127.0.0.1:6826/65552182,v1:127.0.0.1:6827/65552182] [v2:127.0.0.1:6828/65552182,v1:127.0.0.1:6829/65552182] exists,up 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:09:43.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:43.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:09:43.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:09:44.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:09:44.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:09:44.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:09:44.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:09:44.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:09:44.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:09:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:09:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:09:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:09:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T05:09:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:09:44.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:44.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:09:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-02-05T05:09:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-02-05T05:09:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-02-05T05:09:44.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:44.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:09:44.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=231928233990 2024-02-05T05:09:44.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 231928233990 2024-02-05T05:09:44.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-231928233990' 2024-02-05T05:09:44.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:44.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:09:44.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509472 2024-02-05T05:09:44.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509472 2024-02-05T05:09:44.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-231928233990 2-64424509472' 2024-02-05T05:09:44.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:44.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:09:44.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037762 2024-02-05T05:09:44.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037762 2024-02-05T05:09:44.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-231928233990 2-64424509472 3-257698037762' 2024-02-05T05:09:44.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:44.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-02-05T05:09:44.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:44.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:09:44.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-02-05T05:09:44.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:44.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-02-05T05:09:44.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-02-05T05:09:44.811 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836524 2024-02-05T05:09:44.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:45.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836524 2024-02-05T05:09:45.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:09:46.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:09:46.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:46.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836524 2024-02-05T05:09:46.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:09:47.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:09:47.352 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:47.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836524 2024-02-05T05:09:47.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:47.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-231928233990 2024-02-05T05:09:47.633 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:47.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:09:47.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-231928233990 2024-02-05T05:09:47.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:47.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=231928233990 2024-02-05T05:09:47.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 231928233990' 2024-02-05T05:09:47.635 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 231928233990 2024-02-05T05:09:47.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:47.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 231928233990 -lt 231928233990 2024-02-05T05:09:47.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:47.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509472 2024-02-05T05:09:47.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:47.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:09:47.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509472 2024-02-05T05:09:47.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:47.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509472 2024-02-05T05:09:47.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-02-05T05:09:47.906 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509472 2024-02-05T05:09:47.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:09:48.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509473 -lt 64424509472 2024-02-05T05:09:48.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:48.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037762 2024-02-05T05:09:48.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:48.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:09:48.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037762 2024-02-05T05:09:48.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:48.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037762 2024-02-05T05:09:48.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037762' 2024-02-05T05:09:48.175 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 257698037762 2024-02-05T05:09:48.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:09:48.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037762 -lt 257698037762 2024-02-05T05:09:48.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:09:48.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:48.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:48.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:09:48.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:09:48.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:09:48.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:09:48.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:09:48.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:09:48.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:09:48.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:09:49.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:09:49.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:09:49.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:09:49.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:09:49.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:09:49.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:09:49.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:09:49.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:09:49.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:09:49.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191260"' 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191260' 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191262"' 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191262' 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 191260 191262' 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191263: /' 2024-02-05T05:09:49.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:09:49.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191260 2024-02-05T05:09:49.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:09:49.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191265: /' 2024-02-05T05:09:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:09:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:09:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:09:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:09:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:09:52.372 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:09:52.373 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:09:52.374 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:09:52.374 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:09:52.374 INFO:tasks.workunit.client.0.smithi188.stderr:191263: start osd.1 2024-02-05T05:09:52.374 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:09:52.422 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:09:52.423 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:09:52.424 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:09:52.424 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:09:52.424 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:09:52.424 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:09:52.424 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:09:52.424 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:09:52.424 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:09:52.425 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:09:52.426 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:09:52.427 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:09:52.427 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:09:52.427 INFO:tasks.workunit.client.0.smithi188.stderr:191265: start osd.2 2024-02-05T05:09:52.427 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:09:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:09:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:09:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:09:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:09:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:09:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 2024-02-05T05:09:52.450+0000 7fb083d8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 2024-02-05T05:09:52.457+0000 7fb083d8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 2024-02-05T05:09:52.458+0000 7fb083d8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 0 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 2024-02-05T05:09:52.980+0000 7fb083d8d4c0 -1 Falling back to public interface 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 1 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 2 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 2024-02-05T05:09:55.830+0000 7fb083d8d4c0 -1 osd.2 62 log_to_monitors true 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 3 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191265: osd.2 up in weight 1 up_from 69 up_thru 50 down_at 63 last_191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 2024-02-05T05:09:52.398+0000 7f99b77374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 2024-02-05T05:09:52.401+0000 7f99b77374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 2024-02-05T05:09:52.402+0000 7f99b77374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 0 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 2024-02-05T05:09:52.917+0000 7f99b77374c0 -1 Falling back to public interface 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 1 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 2 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:09:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 2024-02-05T05:09:55.644+0000 7f99b77374c0 -1 osd.1 62 log_to_monitors true 2024-02-05T05:09:57.032 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:09:57.032 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:09:57.032 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:09:57.032 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 3 2024-02-05T05:09:57.032 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:09:57.032 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:09:57.995 INFO:tasks.workunit.client.0.smithi188.stderr:191263: osd.1 up in weight 1 up_from 68 up_thru 57 down_at 63 last_clean_interval [15,62) [v2:127.0.0.1:6818/267573689,v1:127.0.0.1:6819/267573689] [v2:127.0.0.1:6820/267573689,v1:127.0.0.1:6821/267573689] exists,up 60c7027b-262c-4328-bdf9-a5c87f3e89d2 2024-02-05T05:09:57.995 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:09:57.996 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 1 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 2 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: 3' 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836528 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836528 2024-02-05T05:09:57.997 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528' 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-292057776130' 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tellclean_interval [54,62) [v2:127.0.0.1:6810/3798524037,v1:127.0.0.1:6811/3798524037] [v2:127.0.0.1:6812/3798524037,v1:127.0.0.1:6813/3798524037] exists,up 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:09:57.998 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 1 2024-02-05T05:09:57.999 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 2 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: 3' 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776131 2024-02-05T05:09:58.000 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776131 2024-02-05T05:09:58.001 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-292057776131' 2024-02-05T05:09:58.001 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:59.119 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:09:59.119 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743426 2024-02-05T05:09:59.119 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743426 2024-02-05T05:09:59.119 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-292057776131 2-296352743426' 2024-02-05T05:09:59.119 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037766 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037766 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-292057776131 2-296352743426 3-257698037766' 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-02-05T05:09:59.120 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: waiting osd.0 seq 21474836527 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836527 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776131 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776131 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776131 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2024-02-05T05:09:59.121 INFO:tasks.workunit.client.0.smithi188.stderr:191263: waiting osd.1 seq 292057776131 2024-02-05T05:09:59.122 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:59.122 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776131 2024-02-05T05:09:59.122 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:59.122 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743426 2024-02-05T05:09:59.122 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:59.122 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:09:59.122 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743426 2024-02-05T05:09:59.123 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/ceph osd.2 flush_pg_stats 2024-02-05T05:09:59.123 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=296352743427 2024-02-05T05:09:59.123 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 296352743427 2024-02-05T05:09:59.123 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-292057776130 2-296352743427' 2024-02-05T05:09:59.123 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037767 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037767 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-292057776130 2-296352743427 3-257698037767' 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836528 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836528 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:59.124 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836528 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: waiting osd.0 seq 21474836528 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836528 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776130 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776130 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-02-05T05:09:59.125 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: waiting osd.1 seq 292057776130 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776130 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-296352743427 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:09:59.126 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-296352743427 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743427 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: waiting osd.2 seq 296352743427 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743427 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037767 2024-02-05T05:10:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037767 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037767 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037767' 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: waiting osd.3 seq 257698037767 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037767 -lt 257698037767 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:00.679 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:10:00.680 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:10:00.806 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/cetest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=296352743426 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: waiting osd.2 seq 296352743426 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 296352743427 -lt 296352743426 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037766 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037766 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037766 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037766' 2024-02-05T05:10:00.807 INFO:tasks.workunit.client.0.smithi188.stderr:191263: waiting osd.3 seq 257698037766 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037767 -lt 257698037766 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:00.808 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:00.809 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:10:00.809 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:00.809 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:00.809 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:00.809 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:10:00.809 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalonph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:01.954 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:01.955 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:01.956 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:01.956 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:01.956 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:01.956 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:01.956 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:01.956 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:02.094 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653:e/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:02.094 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-02-05T05:10:02.094 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:02.094 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:02.094 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:02.095 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:02.096 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:03.340 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1 wait_for_clean: true 2024-02-05T05:10:03.340 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:03.340 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:03.341 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:03.342 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:03.343 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:03.343 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:03.484 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='sele653: wait_for_clean: true 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 5 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:03.486 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:03.487 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:03.487 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:03.487 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:04.731 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='ct(contains("stale") | not)' 2024-02-05T05:10:04.731 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:04.731 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:04.731 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:10:04.731 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:04.731 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:04.732 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:04.733 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:04.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sselect(contains("stale") | not)' 2024-02-05T05:10:04.881 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:04.881 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 2 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:04.882 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:191263: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:10:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191262 2024-02-05T05:10:05.880 INFO:tasks.workunit.client.0.smithi188.stderr:ec 2024-02-05T05:10:05.880 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:191265: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:191265: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:10:05.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:10:05.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:10:05.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:10:05.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:10:06.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:10:06.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:10:06.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:10:06.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:10:06.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:06.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:06.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:06.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:06.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:09:34.182940+0000 2024-02-05T05:10:06.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:10:06.671 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:10:06.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:09:34.182940+0000 2024-02-05T05:10:06.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:10:06.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:09:34.182940+0000 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:06.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:06.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:34.182940+0000 '>' 2024-02-05T05:09:34.182940+0000 2024-02-05T05:10:06.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:07.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:07.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:07.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:10:07.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:07.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:07.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:07.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:08.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:09:34.182940+0000 '>' 2024-02-05T05:09:34.182940+0000 2024-02-05T05:10:08.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:09.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:09.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:09.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:10:09.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:09.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:09.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:09.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:09.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:06.975305+0000 '>' 2024-02-05T05:09:34.182940+0000 2024-02-05T05:10:09.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:10:09.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194579"' 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194579' 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194581"' 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194581' 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:10:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 194579 194581' 2024-02-05T05:10:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:10:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:10:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194579 2024-02-05T05:10:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194582: /' 2024-02-05T05:10:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:10:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194584: /' 2024-02-05T05:10:12.020 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.020 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:12.020 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:10:12.020 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:12.020 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: _ 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: hinfo_key 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: snapset 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:10:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:12.023 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:12.023 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:12.023 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:12.023 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:10:12.023 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:12.024 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:10:12.025 INFO:tasks.workunit.client.0.smithi188.stderr:194584: start osd.2 2024-02-05T05:10:12.049 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.049 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:12.049 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:10:12.049 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:12.049 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:12.050 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: _ 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: hinfo_key 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: snapset 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:10:12.051 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:12.052 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:12.053 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:10:12.054 INFO:tasks.workunit.client.0.smithi188.stderr:194582: start osd.1 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 2024-02-05T05:10:12.052+0000 7f94fc76e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 2024-02-05T05:10:12.055+0000 7f94fc76e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 2024-02-05T05:10:12.057+0000 7f94fc76e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:16.687 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 0 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 2024-02-05T05:10:12.571+0000 7f94fc76e4c0 -1 Falling back to public interface 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 1 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 2 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 2024-02-05T05:10:15.306+0000 7f94fc76e4c0 -1 osd.2 71 log_to_monitors true 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.689 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 3 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194584: osd.2 up in weight 1 up_from 77 up_thru 50 down_aovery_ops 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 2024-02-05T05:10:12.080+0000 7fb3f360c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 2024-02-05T05:10:12.083+0000 7fb3f360c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 2024-02-05T05:10:12.084+0000 7fb3f360c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:16.690 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 0 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 2024-02-05T05:10:12.606+0000 7fb3f360c4c0 -1 Falling back to public interface 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 1 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.691 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 2 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 2024-02-05T05:10:15.449+0000 7fb3f360c4c0 -1 osd.1 71 log_to_monitors true 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 3 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:16.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:17.542 INFO:tasks.workunit.client.0.smithi188.stderr:194582: osd.1 up in weight 1 up_from 77 up_thru 69 down_at 72 last_clean_interval [69,71) [v2:127.0.0.1:6810/2787953153,v1:127.0.0.1:6811/2787953153] [v2:127.0.0.1:6812/2787953153,v1:127.0.0.1:6813/2787953153] exists,up 60c7027b-262c-4328-bdf9-a5c87f3e89d2 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:17.543 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 1 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 2 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: 3' 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-02-05T05:10:17.544 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-330712481794' 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 72 last_clean_interval [68,71) [v2:127.0.0.1:6818/368811872,v1:127.0.0.1:6819/368811872] [v2:127.0.0.1:6820/368811872,v1:127.0.0.1:6821/368811872] exists,up 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:17.545 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 1 2024-02-05T05:10:17.546 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 2 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: 3' 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836533 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836533 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533' 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-330712481795' 2024-02-05T05:10:17.547 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ts: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-330712481794 2-330712481794' 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037773 2024-02-05T05:10:19.869 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037773 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-330712481794 2-330712481794 3-257698037773' 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: waiting osd.0 seq 21474836534 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836534 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836534 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: waiting osd.1 seq 330712481794 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-02-05T05:10:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481795 2024-02-05T05:10:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481795 2024-02-05T05:10:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-330712481795 2-330712481795' 2024-02-05T05:10:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:10:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037772 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037772 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-330712481795 2-330712481795 3-257698037772' 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836533 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836533 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836533 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: waiting osd.0 seq 21474836533 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836533 2024-02-05T05:10:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836533 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481795 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481795 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481795' 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: waiting osd.1 seq 330712481795 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:19.921 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-02-05T05:10:21.058 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seq$seqs 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481794 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481794 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481794' 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: waiting osd.2 seq 330712481794 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037773 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037773 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037773 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037773' 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: waiting osd.3 seq 257698037773 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037773 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:21.060 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:21.061 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:21.173 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtes 2024-02-05T05:10:21.173 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481795 2024-02-05T05:10:21.173 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:21.173 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:21.173 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481795 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481795 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481795' 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: waiting osd.2 seq 330712481795 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481795 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-257698037772 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:21.174 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-257698037772 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037772 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 257698037772' 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: waiting osd.3 seq 257698037772 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037773 -lt 257698037772 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:21.175 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:21.176 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:21.177 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:22.496 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/cst/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:22.497 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:22.498 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:22.499 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:22.499 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:22.499 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:22.499 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:22.601 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntulone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:22.601 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:22.601 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-02-05T05:10:22.601 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-02-05T05:10:22.601 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:22.601 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:22.602 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 5 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:22.603 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cep/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:23.692 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 1 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:23.693 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:23.694 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.clienthtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:23.839 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 1 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:23.840 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:23.841 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:23.841 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:23.841 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:23.841 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:10:23.841 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:23.841 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:25.200 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/q.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:25.200 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:10:25.200 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:25.201 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:194584: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:a/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:194582: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:194582: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194581 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:10:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:10:25.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:10:25.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:10:25.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-02-05T05:10:25.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:10:25.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:10:25.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:10:25.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197746"' 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197746' 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197748"' 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197748' 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:10:25.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 197746 197748' 2024-02-05T05:10:25.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:10:25.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197749: /' 2024-02-05T05:10:25.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:10:25.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197746 2024-02-05T05:10:25.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:10:25.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197751: /' 2024-02-05T05:10:28.106 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.106 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:28.107 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:28.108 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:28.109 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:28.109 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:28.109 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:28.110 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: start osd.1 2024-02-05T05:10:28.111 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:10:28.469 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:28.470 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:28.471 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:28.472 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:10:28.474 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:10:28.474 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:10:28.474 INFO:tasks.workunit.client.0.smithi188.stderr:197749: start osd.3 2024-02-05T05:10:28.474 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 2024-02-05T05:10:28.136+0000 7f99043124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 2024-02-05T05:10:28.139+0000 7f99043124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 2024-02-05T05:10:28.140+0000 7f99043124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:10:32.718 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 0 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 2024-02-05T05:10:28.660+0000 7f99043124c0 -1 Falling back to public interface 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 1 2024-02-05T05:10:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 2 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 2024-02-05T05:10:31.545+0000 7f99043124c0 -1 osd.1 80 log_to_monitors true 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:32.720 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 3 2024-02-05T05:10:32.721 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.721 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197751: osd.1 up in weight 1 up_from 85 up_thru 77 down_at 81 last_197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 2024-02-05T05:10:28.496+0000 7fa947d104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 2024-02-05T05:10:28.535+0000 7fa947d104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 2024-02-05T05:10:28.536+0000 7fa947d104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:32.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 0 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 2024-02-05T05:10:29.063+0000 7fa947d104c0 -1 Falling back to public interface 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 1 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 2 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 2024-02-05T05:10:31.847+0000 7fa947d104c0 -1 osd.3 80 log_to_monitors true 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:32.941 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 3 2024-02-05T05:10:32.942 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:32.942 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:33.569 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [77,80) [v2:127.0.0.1:6818/145539985,v1:127.0.0.1:6819/145539985] [v2:127.0.0.1:6820/145539985,v1:127.0.0.1:6821/145539985] exists,up 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:10:33.569 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:33.569 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:33.570 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 1 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 2 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: 3' 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836538 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836538 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538' 2024-02-05T05:10:33.571 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:33.572 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:33.572 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220162 2024-02-05T05:10:33.572 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220162 2024-02-05T05:10:33.572 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-365072220162' 2024-02-05T05:10:33.572 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:34.937 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tells.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:34.937 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:34.937 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:34.937 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:10:34.937 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 4 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: osd.3 up in weight 1 up_from 89 up_thru 89 down_at 81 last_clean_interval [60,80) [v2:127.0.0.1:6826/3495410343,v1:127.0.0.1:6827/3495410343] [v2:127.0.0.1:6828/3495410343,v1:127.0.0.1:6829/3495410343] exists,up 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:34.938 INFO:tasks.workunit.client.0.smithi188.stderr:197749: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:34.939 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 1 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 2 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: 3' 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-02-05T05:10:34.940 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:36.039 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hel osd.2 flush_pg_stats 2024-02-05T05:10:36.039 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481799 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481799 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-365072220162 2-330712481799' 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836538 1-365072220162 2-330712481799 3-382252089346' 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836538 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836538 2024-02-05T05:10:36.040 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836538 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: waiting osd.0 seq 21474836538 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836538 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836538 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-365072220162 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-365072220162 2024-02-05T05:10:36.041 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:36.042 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220162 2024-02-05T05:10:36.042 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 365072220162' 2024-02-05T05:10:36.042 INFO:tasks.workunit.client.0.smithi188.stderr:197751: waiting osd.1 seq 365072220162 2024-02-05T05:10:36.042 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:36.042 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220162 -lt 365072220162 2024-02-05T05:10:36.042 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197751: pers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=365072220163 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 365072220163 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-365072220163' 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481800 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481800 2024-02-05T05:10:36.881 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-365072220163 2-330712481800' 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089347 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089347 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-365072220163 2-330712481800 3-382252089347' 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-02-05T05:10:36.882 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: waiting osd.0 seq 21474836540 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836540 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836540 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-365072220163 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-365072220163 2024-02-05T05:10:36.883 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:37.313 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/u//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481799 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481799 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481799 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481799' 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: waiting osd.2 seq 330712481799 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481799 -lt 330712481799 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089346 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:37.314 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089346 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089346' 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: waiting osd.3 seq 382252089346 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089346 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:37.315 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:10:37.316 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:38.115 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.clienbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=365072220163 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 365072220163' 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: waiting osd.1 seq 365072220163 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 365072220163 -lt 365072220163 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481800 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481800 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481800 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481800' 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: waiting osd.2 seq 330712481800 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481801 -lt 330712481800 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089347 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089347 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089347 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089347' 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: waiting osd.3 seq 382252089347 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089347 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:38.118 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:38.119 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:38.119 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:38.119 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:38.119 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:38.119 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("t.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:38.794 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:10:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:10:39.543 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clonstale") | not)' 2024-02-05T05:10:39.544 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:39.544 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:39.544 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:10:39.544 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:39.545 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:197749: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:10:39.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197748 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:e.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:39.684 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:197751: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:197751: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:10:39.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:10:39.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:10:39.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:10:39.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:10:39.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:10:39.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:10:39.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:10:39.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:39.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:39.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:39.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:40.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:10:06.975305+0000 2024-02-05T05:10:40.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:10:40.504 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:10:40.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:10:06.975305+0000 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:10:06.975305+0000 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:40.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:40.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:06.975305+0000 '>' 2024-02-05T05:10:06.975305+0000 2024-02-05T05:10:40.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:41.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:41.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:41.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:10:41.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:41.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:41.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:41.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:42.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:06.975305+0000 '>' 2024-02-05T05:10:06.975305+0000 2024-02-05T05:10:42.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:43.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:43.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:43.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:10:43.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:43.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:43.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:43.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:43.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:06.975305+0000 '>' 2024-02-05T05:10:06.975305+0000 2024-02-05T05:10:43.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:10:44.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:10:44.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:10:44.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:10:44.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:10:44.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:10:44.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:10:44.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:10:44.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:10:40.794323+0000 '>' 2024-02-05T05:10:06.975305+0000 2024-02-05T05:10:44.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:10:44.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:10:44.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:10:44.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200769"' 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200769' 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200771"' 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200771' 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 200769 200771' 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:10:44.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200769 2024-02-05T05:10:44.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200772: /' 2024-02-05T05:10:44.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:44.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200774: /' 2024-02-05T05:10:47.259 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:10:47.260 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:10:47.261 INFO:tasks.workunit.client.0.smithi188.stderr:200772: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: _ 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: hinfo_key 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: snapset 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:47.262 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:200772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:10:47.264 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:10:47.265 INFO:tasks.workunit.client.0.smithi188.stderr:200772: start osd.3 2024-02-05T05:10:47.596 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.596 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:10:47.596 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:10:47.596 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:10:47.596 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:47.596 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.596 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:10:47.597 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: _ 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: hinfo_key 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: snapset 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:47.598 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:47.599 INFO:tasks.workunit.client.0.smithi188.stderr:200774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:10:47.600 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:10:47.601 INFO:tasks.workunit.client.0.smithi188.stderr:200774: start osd.1 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 2024-02-05T05:10:47.290+0000 7f5c61a134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 2024-02-05T05:10:47.292+0000 7f5c61a134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 2024-02-05T05:10:47.293+0000 7f5c61a134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:51.371 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 0 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 2024-02-05T05:10:47.808+0000 7f5c61a134c0 -1 Falling back to public interface 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:51.372 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 1 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 2 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 2024-02-05T05:10:50.199+0000 7f5c61a134c0 -1 osd.3 91 log_to_monitors true 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:51.374 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:51.374 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:51.374 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 3 2024-02-05T05:10:51.374 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 2024-02-05T05:10:47.624+0000 7fd53b2164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 2024-02-05T05:10:47.627+0000 7fd53b2164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 2024-02-05T05:10:47.628+0000 7fd53b2164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 0 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 2024-02-05T05:10:48.152+0000 7fd53b2164c0 -1 Falling back to public interface 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:52.249 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 1 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 2 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 2024-02-05T05:10:50.844+0000 7fd53b2164c0 -1 osd.1 91 log_to_monitors true 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:10:52.250 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:10:52.251 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:10:52.251 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:10:52.251 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 3 2024-02-05T05:10:52.251 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:10:52.251 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:10:52.614 INFO:tasks.workunit.client.0.smithi188.stderr:200774: osd.1 up in weight 1 up_from 98 up_thru 85 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: osd.3 up in weight 1 up_from 96 up_thru 96 down_at 92 last_clean_interval [89,91) [v2:127.0.0.1:6818/4006974270,v1:127.0.0.1:6819/4006974270] [v2:127.0.0.1:6820/4006974270,v1:127.0.0.1:6821/4006974270] exists,up 70f33e81-0a0a-4d5a-9f2b-8c6130c15a47 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:52.616 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:52.619 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:52.619 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:52.619 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:52.619 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 1 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 2 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: 3' 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-02-05T05:10:52.620 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:52.621 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:52.621 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-02-05T05:10:52.621 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-02-05T05:10:52.621 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-420906795010' 2024-02-05T05:10:53.208 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 92 last_clean_interval [85,91) [v2:127.0.0.1:6826/3162752881,v1:127.0.0.1:6827/3162752881] [v2:127.0.0.1:6828/3162752881,v1:127.0.0.1:6829/3162752881] exists,up 73d76fcf-c4a3-454e-a5f0-53de684d4cc0 2024-02-05T05:10:53.208 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:10:53.208 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:10:53.209 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 1 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 2 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: 3' 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836545 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836545 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545' 2024-02-05T05:10:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:53.211 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:10:53.211 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-02-05T05:10:53.211 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-02-05T05:10:53.211 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-420906795011' 2024-02-05T05:10:53.211 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stan $ids 2024-02-05T05:10:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481805 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481805 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-420906795010 2-330712481805' 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-420906795010 2-330712481805 3-412316860418' 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-02-05T05:10:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: waiting osd.0 seq 21474836544 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836544 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-420906795010 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-420906795010 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 420906795010' 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: waiting osd.1 seq 420906795010 2024-02-05T05:10:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:54.835 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795010 2024-02-05T05:10:54.835 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:54.835 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:54.835 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:54.835 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795010 2024-02-05T05:10:55.313 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/uts: ceph tell osd.2 flush_pg_stats 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481806 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481806 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-420906795011 2-330712481806' 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-420906795011 2-330712481806 3-412316860419' 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836545 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:55.314 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836545 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836545 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: waiting osd.0 seq 21474836545 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836545 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:10:55.315 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836545 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-420906795011 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-420906795011 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 420906795011' 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: waiting osd.1 seq 420906795011 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:10:55.316 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795011 2024-02-05T05:10:56.058 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in buntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481805 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481805 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481805 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481805' 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: waiting osd.2 seq 330712481805 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481806 -lt 330712481805 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-412316860418 2024-02-05T05:10:56.059 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-412316860418 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 412316860418' 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: waiting osd.3 seq 412316860418 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860418 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:56.060 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:56.061 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:56.061 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:56.061 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:56.061 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:56.061 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:56.061 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:56.061 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:56.422 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:200772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:200772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:10:56.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200771 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:$seqs 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-330712481806 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-330712481806 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481806 2024-02-05T05:10:56.546 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 330712481806' 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: waiting osd.2 seq 330712481806 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481806 -lt 330712481806 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-412316860419 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-412316860419 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 412316860419' 2024-02-05T05:10:56.547 INFO:tasks.workunit.client.0.smithi188.stderr:200774: waiting osd.3 seq 412316860419 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860419 -lt 412316860419 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:10:56.548 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:10:56.549 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:10:56.549 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:10:56.549 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:10:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:10:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:200774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:10:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:10:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:10:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:200774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:10:56.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:10:56.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:10:56.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:10:56.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:10:56.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:10:56.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:10:56.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:10:56.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:10:56.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:10:56.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:10:56.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:10:56.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:10:56.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:56.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:56.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:56.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:56.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:57.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:57.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:10:57.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:10:57.020 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:10:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:10:57.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:10:57.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:10:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:10:57.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:10:57.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:10:57.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:10:57.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:10:57.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:10:57.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:10:57.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:10:57.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:57.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:57.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:10:57.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:10:57.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:10:57.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:10:57.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:10:57.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:10:57.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:10:57.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:10:57.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:10:57.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:10:57.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:10:57.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:10:57.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:10:57.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:10:57.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:10:57.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:10:57.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:10:57.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:10:57.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:10:57.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:57.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:57.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:10:57.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:10:57.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:10:57.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:10:57.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:10:57.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:57.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:57.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:10:57.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:10:57.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:10:57.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:10:57.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:10:57.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:10:57.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:10:57.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:10:57.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:10:57.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:10:57.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:57.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:57.427 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:57.427 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:57.427 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:57.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:57.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:10:57.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:10:57.621 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:10:57.622 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:10:57.622 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:57.622 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:57.622 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:10:57.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:10:57.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:10:57.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:10:57.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:10:57.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:10:57.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:10:57.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:10:57.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-02-05T05:10:57.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:10:57.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:10:57.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:10:57.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:10:57.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:10:57.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:10:58.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:10:58.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:58.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:58.597 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:58.597 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:58.597 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:58.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:58.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:10:58.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:10:58.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:10:58.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:10:58.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=750b822c-bde8-42c2-9644-cf05193dbf56 2024-02-05T05:10:58.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 750b822c-bde8-42c2-9644-cf05193dbf56' 2024-02-05T05:10:58.632 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 750b822c-bde8-42c2-9644-cf05193dbf56 2024-02-05T05:10:58.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:10:58.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDibcBlMStyJhAAraT/x2VmTvDGLDHoLAL0kA== 2024-02-05T05:10:58.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDibcBlMStyJhAAraT/x2VmTvDGLDHoLAL0kA=="}' 2024-02-05T05:10:58.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 750b822c-bde8-42c2-9644-cf05193dbf56 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:10:58.853 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:10:58.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:10:58.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDibcBlMStyJhAAraT/x2VmTvDGLDHoLAL0kA== --osd-uuid 750b822c-bde8-42c2-9644-cf05193dbf56 2024-02-05T05:10:58.892 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:10:58.890+0000 7f85d46104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:58.903 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:10:58.902+0000 7f85d46104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:58.904 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:10:58.903+0000 7f85d46104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:10:58.905 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:10:58.903+0000 7f85d46104c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:10:58.905 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:10:58.903+0000 7f85d46104c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:10:58.905 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:10:58.903+0000 7f85d46104c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:10:58.996 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:10:58.994+0000 7f85d46104c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:11:04.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:11:04.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:11:04.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:11:04.509 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:11:04.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:11:04.855 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:11:04.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:11:04.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:11:04.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:11:04.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:11:04.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:11:04.881 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:04.879+0000 7f58564164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:04.883 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:04.881+0000 7f58564164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:04.884 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:04.882+0000 7f58564164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:05.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:11:05.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:11:05.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:11:05.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:11:05.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:11:05.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:05.124 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:11:05.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:11:05.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:05.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:05.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:05.406 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:05.405+0000 7f58564164c0 -1 Falling back to public interface 2024-02-05T05:11:06.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:06.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:06.393 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:11:06.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:11:06.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:06.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:06.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:07.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:07.667 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:11:07.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:07.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:11:07.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:07.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:07.780 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:07.778+0000 7f58564164c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:11:07.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:08.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:08.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:08.964 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:11:08.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:11:08.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:08.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:09.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:10.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:10.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:10.239 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:11:10.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:11:10.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:10.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:10.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:11.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:11.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:11.593 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:11:11.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:11:11.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:11.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:11.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:12.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:12.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:12.870 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:11:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:11:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:13.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:14.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:14.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:14.147 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:11:14.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:11:14.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:14.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:14.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:15.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:15.441 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:11:15.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:11:15.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:15.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:15.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:16.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:16.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:16.838 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:11:16.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:11:16.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:16.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:17.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:18.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:18.114 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:11:18.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:18.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:11:18.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:18.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:18.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:19.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:19.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:19.465 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:11:19.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:11:19.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:19.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:19.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:20.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:20.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:20.802 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:11:20.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:11:20.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:20.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:21.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:22.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:22.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:22.683 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:11:22.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:11:22.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:22.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:23.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:24.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:24.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:24.617 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:11:24.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:11:24.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:24.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:25.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:26.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:26.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:26.036 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:11:26.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:11:26.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:26.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:26.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:27.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:27.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:27.814 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:11:27.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:11:27.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:27.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:28.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:29.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:29.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:29.534 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:11:29.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:11:29.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:29.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:30.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:31.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:31.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:31.279 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:11:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:11:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:11:31.553 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3086230965,v1:127.0.0.1:6803/3086230965] [v2:127.0.0.1:6804/3086230965,v1:127.0.0.1:6805/3086230965] exists,up 750b822c-bde8-42c2-9644-cf05193dbf56 2024-02-05T05:11:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:11:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:11:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:11:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:11:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-02-05T05:11:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:11:31.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:11:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fd26eaf4-2b3d-4944-9fac-679f115065e3 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fd26eaf4-2b3d-4944-9fac-679f115065e3' 2024-02-05T05:11:31.561 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 fd26eaf4-2b3d-4944-9fac-679f115065e3 2024-02-05T05:11:31.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:11:31.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQADbsBlzK87IhAAPuZdh78kH/c6ukO8MP8rag== 2024-02-05T05:11:31.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQADbsBlzK87IhAAPuZdh78kH/c6ukO8MP8rag=="}' 2024-02-05T05:11:31.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fd26eaf4-2b3d-4944-9fac-679f115065e3 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:11:31.928 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:11:31.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:11:31.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADbsBlzK87IhAAPuZdh78kH/c6ukO8MP8rag== --osd-uuid fd26eaf4-2b3d-4944-9fac-679f115065e3 2024-02-05T05:11:31.968 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:31.966+0000 7ff0beab64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:32.032 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:32.030+0000 7ff0beab64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:32.033 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:32.031+0000 7ff0beab64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:32.033 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:32.031+0000 7ff0beab64c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:11:32.033 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:32.031+0000 7ff0beab64c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:11:32.033 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:32.031+0000 7ff0beab64c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:11:32.113 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:32.111+0000 7ff0beab64c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:11:37.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:11:37.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:11:37.980 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:11:37.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:11:37.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:11:38.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:11:38.330 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:11:38.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:11:38.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:11:38.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:11:38.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:11:38.355 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:38.354+0000 7fc61740e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:38.357 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:38.356+0000 7fc61740e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:38.358 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:38.357+0000 7fc61740e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:11:38.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:11:38.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:11:38.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:11:38.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:11:38.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:11:38.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:38.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:11:38.624 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:11:38.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:38.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:38.878 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:38.876+0000 7fc61740e4c0 -1 Falling back to public interface 2024-02-05T05:11:38.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:39.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:39.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:39.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:11:39.907 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:11:39.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:39.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:40.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:41.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:41.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:41.181 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:11:41.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:11:41.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:41.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:41.246 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:11:41.244+0000 7fc61740e4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:11:41.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:42.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:42.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:42.464 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:11:42.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:11:42.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:42.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:42.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:43.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:43.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:43.787 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:11:43.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:11:43.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:43.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:44.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:45.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:45.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:45.127 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:11:45.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:11:45.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:45.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:45.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:46.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:46.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:46.459 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:11:46.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:11:46.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:46.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:46.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:47.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:47.788 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:11:47.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:47.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:11:47.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:47.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:48.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:49.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:49.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:49.227 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:11:49.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:11:49.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:49.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:49.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:50.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:50.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:50.593 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:11:50.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:11:50.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:50.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:50.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:51.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:51.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:51.940 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:11:51.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:11:51.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:51.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:52.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:53.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:53.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:53.269 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:11:53.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:11:53.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:53.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:53.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:54.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:54.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:54.547 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:11:54.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:11:54.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:54.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:55.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:56.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:56.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:56.353 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:11:56.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:11:56.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:56.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:57.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:58.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:58.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:58.113 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:11:58.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:11:58.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:58.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:11:58.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:11:59.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:11:59.752 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:11:59.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:11:59.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:11:59.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:11:59.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:00.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:01.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:01.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:01.454 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:12:01.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:12:01.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:01.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:01.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:02.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:02.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:02.770 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:12:02.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:12:02.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:02.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:03.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:04.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:04.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:04.531 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:12:04.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:12:04.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:04.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:12:04.826 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1995910996,v1:127.0.0.1:6811/1995910996] [v2:127.0.0.1:6812/1995910996,v1:127.0.0.1:6813/1995910996] exists,up fd26eaf4-2b3d-4944-9fac-679f115065e3 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:12:04.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:12:04.831 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:12:04.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:12:04.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:12:04.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:12:04.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:12:04.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=73a66e85-0b70-4ae9-ad89-e99dcaf05d23 2024-02-05T05:12:04.833 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 73a66e85-0b70-4ae9-ad89-e99dcaf05d23 2024-02-05T05:12:04.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 73a66e85-0b70-4ae9-ad89-e99dcaf05d23' 2024-02-05T05:12:04.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:12:04.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkbsBlg81+MhAAgGI6CN1Pu3iRKZ6HbCpdAg== 2024-02-05T05:12:04.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkbsBlg81+MhAAgGI6CN1Pu3iRKZ6HbCpdAg=="}' 2024-02-05T05:12:04.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 73a66e85-0b70-4ae9-ad89-e99dcaf05d23 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:12:05.242 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:12:05.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:12:05.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkbsBlg81+MhAAgGI6CN1Pu3iRKZ6HbCpdAg== --osd-uuid 73a66e85-0b70-4ae9-ad89-e99dcaf05d23 2024-02-05T05:12:05.282 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:05.280+0000 7f1e5b8124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:05.318 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:05.316+0000 7f1e5b8124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:05.317+0000 7f1e5b8124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:05.318+0000 7f1e5b8124c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:12:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:05.318+0000 7f1e5b8124c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:12:05.320 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:05.318+0000 7f1e5b8124c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:12:05.393 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:05.391+0000 7f1e5b8124c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:12:11.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:12:11.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:12:11.061 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:12:11.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:12:11.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:12:11.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:12:11.405 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:12:11.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:12:11.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:12:11.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:12:11.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:12:11.430 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:11.429+0000 7fefc8c494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:11.432 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:11.431+0000 7fefc8c494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:11.433 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:11.432+0000 7fefc8c494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:11.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:12:11.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:12:11.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:12:11.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:12:11.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:12:11.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:11.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:12:11.675 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:12:11.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:11.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:11.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:11.953 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:11.952+0000 7fefc8c494c0 -1 Falling back to public interface 2024-02-05T05:12:12.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:12.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:12.941 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:12:12.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:12:12.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:12.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:13.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:14.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:14.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:14.213 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:12:14.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:12:14.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:14.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:14.323 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:14.322+0000 7fefc8c494c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:12:14.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:15.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:15.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:15.551 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:12:15.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:12:15.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:15.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:15.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:16.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:16.830 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:12:16.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:16.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:12:16.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:16.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:17.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:18.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:18.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:18.135 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:12:18.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:12:18.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:18.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:18.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:19.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:19.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:19.416 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:12:19.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:12:19.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:19.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:19.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:20.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:20.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:20.695 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:12:20.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:12:20.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:20.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:20.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:21.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:21.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:21.976 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:12:21.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:12:21.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:21.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:22.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:23.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:23.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:23.328 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:12:23.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:12:23.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:23.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:23.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:24.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:24.608 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:12:24.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:12:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:24.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:25.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:25.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:25.952 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:12:25.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:12:25.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:25.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:26.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:27.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:27.453 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:12:27.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:27.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:12:27.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:27.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:28.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:29.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:29.052 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:12:29.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:29.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:12:29.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:29.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:29.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:30.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:30.436 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:12:30.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:30.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:12:30.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:30.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:30.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:31.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:31.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:31.980 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:12:31.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:12:31.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:31.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:32.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:33.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:33.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:33.703 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:12:33.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:12:33.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:33.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:34.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:35.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:35.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:35.353 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:12:35.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:12:35.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:35.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:35.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:36.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:36.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:36.647 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:12:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:12:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:37.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:38.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:38.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:38.251 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:12:38.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:12:38.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:38.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:12:38.525 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2303947541,v1:127.0.0.1:6819/2303947541] [v2:127.0.0.1:6820/2303947541,v1:127.0.0.1:6821/2303947541] exists,up 73a66e85-0b70-4ae9-ad89-e99dcaf05d23 2024-02-05T05:12:38.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:12:38.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:12:38.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:12:38.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:12:38.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-02-05T05:12:38.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:12:38.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:12:38.530 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:12:38.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:12:38.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:12:38.532 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:12:38.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e670ed6-5780-4b95-8c34-207ce6eba772 2024-02-05T05:12:38.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3e670ed6-5780-4b95-8c34-207ce6eba772' 2024-02-05T05:12:38.532 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 3e670ed6-5780-4b95-8c34-207ce6eba772 2024-02-05T05:12:38.532 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:12:38.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGbsBl3bh/IBAALe0LZEGt3f0ToUnxj7Og+A== 2024-02-05T05:12:38.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGbsBl3bh/IBAALe0LZEGt3f0ToUnxj7Og+A=="}' 2024-02-05T05:12:38.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e670ed6-5780-4b95-8c34-207ce6eba772 -i td/osd-scrub-repair/3/new.json 2024-02-05T05:12:38.875 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:12:38.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:12:38.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGbsBl3bh/IBAALe0LZEGt3f0ToUnxj7Og+A== --osd-uuid 3e670ed6-5780-4b95-8c34-207ce6eba772 2024-02-05T05:12:38.914 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:38.913+0000 7ff8820114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:38.959 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:38.957+0000 7ff8820114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:38.960 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:38.958+0000 7ff8820114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:38.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:38.959+0000 7ff8820114c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:12:38.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:38.959+0000 7ff8820114c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:12:38.964 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:38.959+0000 7ff8820114c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:12:39.039 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:39.037+0000 7ff8820114c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:12:45.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:12:45.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:12:45.091 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T05:12:45.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:12:45.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:12:45.438 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T05:12:45.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:12:45.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:12:45.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:12:45.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:12:45.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:12:45.463 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:45.462+0000 7f73062a54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:45.466 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:45.464+0000 7f73062a54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:45.466 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:45.465+0000 7f73062a54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:12:45.710 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:12:45.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:45.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:45.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:45.990 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:45.988+0000 7f73062a54c0 -1 Falling back to public interface 2024-02-05T05:12:46.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:46.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:46.977 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:12:46.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:12:46.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:46.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:47.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:48.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:48.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:48.254 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:12:48.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:12:48.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:48.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:48.317 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:12:48.315+0000 7f73062a54c0 -1 osd.3 0 log_to_monitors true 2024-02-05T05:12:48.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:49.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:49.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:49.534 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:12:49.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:12:49.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:49.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:49.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:50.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:50.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:50.813 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:12:50.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:12:50.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:50.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:51.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:52.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:52.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:52.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:12:52.137 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:12:52.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:52.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:52.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:53.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:53.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:53.414 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:12:53.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:12:53.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:53.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:53.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:54.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:54.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:54.693 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:12:54.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:12:54.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:54.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:55.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:56.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:56.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:56.097 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:12:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:12:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:56.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:57.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:57.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:57.378 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:12:57.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:12:57.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:57.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:57.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:58.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:58.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:58.658 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:12:58.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:12:58.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:58.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:12:58.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:12:59.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:12:59.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:12:59.963 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:12:59.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:12:59.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:12:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:00.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:01.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:01.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:01.326 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:13:01.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:13:01.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:01.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:01.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:13:02.653 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:13:02.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:02.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:03.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:04.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:04.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:04.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:13:04.106 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:13:04.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:04.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:04.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:05.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:05.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:05.794 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:13:05.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:13:05.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:05.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:06.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:07.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:07.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:07.588 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:13:07.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:13:07.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:07.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:08.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:09.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:09.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:09.333 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:13:09.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:13:09.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:09.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:10.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:11.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:11.103 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:13:11.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:11.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:13:11.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:11.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:13:11.542 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4089471008,v1:127.0.0.1:6827/4089471008] [v2:127.0.0.1:6828/4089471008,v1:127.0.0.1:6829/4089471008] exists,up 3e670ed6-5780-4b95-8c34-207ce6eba772 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:13:11.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:13:11.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e722ff5c-c48d-4a7e-934c-0cc7a3b94ebf 2024-02-05T05:13:11.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 e722ff5c-c48d-4a7e-934c-0cc7a3b94ebf' 2024-02-05T05:13:11.550 INFO:tasks.workunit.client.0.smithi188.stdout:add osd4 e722ff5c-c48d-4a7e-934c-0cc7a3b94ebf 2024-02-05T05:13:11.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:13:11.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnbsBlg/B+IRAApweIntbgnGVsduCwdFFceQ== 2024-02-05T05:13:11.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnbsBlg/B+IRAApweIntbgnGVsduCwdFFceQ=="}' 2024-02-05T05:13:11.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e722ff5c-c48d-4a7e-934c-0cc7a3b94ebf -i td/osd-scrub-repair/4/new.json 2024-02-05T05:13:11.881 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:13:11.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-02-05T05:13:11.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnbsBlg/B+IRAApweIntbgnGVsduCwdFFceQ== --osd-uuid e722ff5c-c48d-4a7e-934c-0cc7a3b94ebf 2024-02-05T05:13:11.920 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:11.919+0000 7f929e3924c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:11.965 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:11.964+0000 7f929e3924c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:11.966 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:11.965+0000 7f929e3924c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:11.967 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:11.965+0000 7f929e3924c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:13:11.967 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:11.965+0000 7f929e3924c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:13:11.967 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:11.965+0000 7f929e3924c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:13:12.040 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:12.039+0000 7f929e3924c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-02-05T05:13:18.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-02-05T05:13:18.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:13:18.220 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd4 key to auth repository 2024-02-05T05:13:18.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-02-05T05:13:18.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:13:18.594 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.4 2024-02-05T05:13:18.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-02-05T05:13:18.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:13:18.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:13:18.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:13:18.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:13:18.620 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:18.618+0000 7fb216d074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:18.622 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:18.620+0000 7fb216d074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:18.623 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:18.621+0000 7fb216d074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:18.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-02-05T05:13:18.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:13:18.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-02-05T05:13:18.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:13:18.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:13:18.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:18.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:13:18.884 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:13:18.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:18.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:19.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:19.135+0000 7fb216d074c0 -1 Falling back to public interface 2024-02-05T05:13:19.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:20.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:20.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:20.154 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:13:20.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:13:20.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:20.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:20.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:21.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:21.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:21.434 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:13:21.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:13:21.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:21.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:21.462 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:21.460+0000 7fb216d074c0 -1 osd.4 0 log_to_monitors true 2024-02-05T05:13:21.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:22.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:22.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:22.839 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:13:22.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:13:22.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:22.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:23.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:24.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:24.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:24.331 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:13:24.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:13:24.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:24.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:25.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:25.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:25.613 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:13:25.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:13:25.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:25.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:25.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:26.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:26.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:26.893 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:13:26.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:13:26.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:26.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:27.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:28.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:28.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:28.209 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:13:28.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:13:28.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:28.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:28.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:29.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:29.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:29.537 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:13:29.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:13:29.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:29.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:29.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:30.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:30.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:30.815 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:13:30.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:13:30.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:30.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:31.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:32.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:32.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:32.175 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:13:32.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:13:32.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:32.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:32.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:33.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:33.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:33.539 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:13:33.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:13:33.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:33.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:33.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:34.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:34.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:34.820 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:13:34.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:13:34.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:34.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:35.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:36.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:36.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:36.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:13:36.566 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:13:36.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:36.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:37.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:38.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:38.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:38.421 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:13:38.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:13:38.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:38.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:39.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:40.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:40.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:40.180 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:13:40.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:13:40.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:40.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:40.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:41.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:41.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:41.967 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:13:41.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:13:41.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:41.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:42.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:43.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:43.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:43.404 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:13:43.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:13:43.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:43.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:43.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:44.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:44.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:44.796 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:13:44.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:13:44.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:44.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:45.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:46.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:46.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:46.078 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:13:46.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:13:46.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:46.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:13:46.350 INFO:tasks.workunit.client.0.smithi188.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3522562006,v1:127.0.0.1:6835/3522562006] [v2:127.0.0.1:6836/3522562006,v1:127.0.0.1:6837/3522562006] exists,up e722ff5c-c48d-4a7e-934c-0cc7a3b94ebf 2024-02-05T05:13:46.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:13:46.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:13:46.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:13:46.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:13:46.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-02-05T05:13:46.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:13:46.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:13:46.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:13:46.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:13:46.357 INFO:tasks.workunit.client.0.smithi188.stdout:add osd5 a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:13:46.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a0f6513a-323f-47de-9504-1ea4e7d0594f' 2024-02-05T05:13:46.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:13:46.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKbsBlFnAgFhAAVqW6GWnjqMmgqF9sGVYucQ== 2024-02-05T05:13:46.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKbsBlFnAgFhAAVqW6GWnjqMmgqF9sGVYucQ=="}' 2024-02-05T05:13:46.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0f6513a-323f-47de-9504-1ea4e7d0594f -i td/osd-scrub-repair/5/new.json 2024-02-05T05:13:46.728 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:13:46.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-02-05T05:13:46.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKbsBlFnAgFhAAVqW6GWnjqMmgqF9sGVYucQ== --osd-uuid a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:13:46.768 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:46.766+0000 7f6f0560e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:46.804 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:46.803+0000 7f6f0560e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:46.805 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:46.804+0000 7f6f0560e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:46.806 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:46.804+0000 7f6f0560e4c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:13:46.806 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:46.804+0000 7f6f0560e4c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:13:46.806 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:46.804+0000 7f6f0560e4c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:13:46.879 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:46.878+0000 7f6f0560e4c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-02-05T05:13:52.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-02-05T05:13:52.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:13:52.615 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd5 key to auth repository 2024-02-05T05:13:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-02-05T05:13:52.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:13:52.995 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.5 2024-02-05T05:13:52.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-02-05T05:13:52.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:13:52.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:13:52.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:13:52.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:13:53.022 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:53.020+0000 7f8fc05204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:53.024 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:53.022+0000 7f8fc05204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:53.025 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:53.023+0000 7f8fc05204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:13:53.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-02-05T05:13:53.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:13:53.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:13:53.265 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:13:53.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:13:53.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:13:53.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:53.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:13:53.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:53.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:13:53.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:53.555 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:53.554+0000 7f8fc05204c0 -1 Falling back to public interface 2024-02-05T05:13:54.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:54.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:54.532 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:13:54.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:13:54.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:54.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:13:54.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:55.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:55.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:55.824 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:13:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:13:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:13:55.878 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:13:55.876+0000 7f8fc05204c0 -1 osd.5 0 log_to_monitors true 2024-02-05T05:13:56.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:57.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:57.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:57.106 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:13:57.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:13:57.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:57.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:13:57.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:58.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:58.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:58.477 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:13:58.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:13:58.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:58.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:13:58.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:13:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:13:59.800 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:13:59.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:13:59.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:13:59.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:13:59.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:00.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:01.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:01.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:01.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:14:01.080 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:14:01.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:01.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:01.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:02.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:02.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:02.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:14:02.362 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:14:02.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:02.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:02.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:03.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:03.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:03.761 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:14:03.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:14:03.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:03.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:04.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:05.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:05.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:05.044 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:14:05.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:14:05.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:05.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:05.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:06.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:06.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:06.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:14:06.322 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:14:06.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:06.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:06.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:07.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:07.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:07.622 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:14:07.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:14:07.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:07.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:07.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:08.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:08.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:08.905 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:14:08.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:14:08.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:08.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:09.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:10.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:10.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:10.658 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:14:10.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:14:10.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:10.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:11.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:12.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:12.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:12.368 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:14:12.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:14:12.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:12.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:12.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:13.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:13.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:13.756 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:14:13.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:14:13.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:13.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:14.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:15.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:15.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:15.348 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:14:15.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:14:15.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:15.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:16.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:17.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:17.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:17.118 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:14:17.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:14:17.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:17.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:17.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:18.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:18.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:18.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:14:18.560 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:14:18.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:18.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:18.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:19.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:19.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:19.919 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:14:19.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:14:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:14:20.194 INFO:tasks.workunit.client.0.smithi188.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2616674815,v1:127.0.0.1:6843/2616674815] [v2:127.0.0.1:6844/2616674815,v1:127.0.0.1:6845/2616674815] exists,up a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:14:20.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:14:20.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:14:20.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:14:20.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:14:20.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-02-05T05:14:20.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:14:20.199 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:14:20.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:14:20.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:14:20.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:14:20.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:14:20.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-02-05T05:14:20.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:14:20.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=156097c6-eff2-4464-a95a-b092a571d27e 2024-02-05T05:14:20.201 INFO:tasks.workunit.client.0.smithi188.stdout:add osd6 156097c6-eff2-4464-a95a-b092a571d27e 2024-02-05T05:14:20.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 156097c6-eff2-4464-a95a-b092a571d27e' 2024-02-05T05:14:20.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:14:20.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsbsBlhHLUDBAASR6cCUFKVlJdCcoqvSondA== 2024-02-05T05:14:20.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsbsBlhHLUDBAASR6cCUFKVlJdCcoqvSondA=="}' 2024-02-05T05:14:20.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 156097c6-eff2-4464-a95a-b092a571d27e -i td/osd-scrub-repair/6/new.json 2024-02-05T05:14:20.561 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:14:20.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-02-05T05:14:20.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsbsBlhHLUDBAASR6cCUFKVlJdCcoqvSondA== --osd-uuid 156097c6-eff2-4464-a95a-b092a571d27e 2024-02-05T05:14:20.600 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:20.599+0000 7f4b532124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:20.657 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:20.656+0000 7f4b532124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:20.658 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:20.657+0000 7f4b532124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:20.659 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:20.657+0000 7f4b532124c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:14:20.659 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:20.657+0000 7f4b532124c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:14:20.659 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:20.657+0000 7f4b532124c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:14:20.723 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:20.721+0000 7f4b532124c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-02-05T05:14:26.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-02-05T05:14:26.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:14:26.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-02-05T05:14:26.962 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd6 key to auth repository 2024-02-05T05:14:26.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:14:27.305 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.6 2024-02-05T05:14:27.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-02-05T05:14:27.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:14:27.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:14:27.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:14:27.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:14:27.331 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:27.329+0000 7fc7c0e9a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:27.333 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:27.331+0000 7fc7c0e9a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:27.334 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:27.332+0000 7fc7c0e9a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:27.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-02-05T05:14:27.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:14:27.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-02-05T05:14:27.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:14:27.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:14:27.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:27.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:14:27.575 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:14:27.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:27.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:27.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:27.851 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:27.850+0000 7fc7c0e9a4c0 -1 Falling back to public interface 2024-02-05T05:14:28.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:28.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:28.844 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:14:28.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:14:28.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:28.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:29.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:30.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:30.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:30.121 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:14:30.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:14:30.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:30.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:30.195 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:30.193+0000 7fc7c0e9a4c0 -1 osd.6 0 log_to_monitors true 2024-02-05T05:14:30.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:31.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:31.399 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:14:31.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:31.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:14:31.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:31.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:31.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:32.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:32.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:32.682 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:14:32.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:14:32.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:32.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:33.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:34.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:34.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:34.012 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:14:34.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:14:34.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:34.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:34.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:35.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:35.291 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:14:35.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:35.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:14:35.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:35.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:35.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:36.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:36.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:36.569 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:14:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:14:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:36.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:37.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:37.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:37.847 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:14:37.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:14:37.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:37.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:38.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:39.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:39.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:39.148 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:14:39.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:14:39.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:39.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:39.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:40.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:40.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:40.425 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:14:40.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:14:40.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:40.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:40.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:41.705 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:14:41.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:14:41.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:41.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:42.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:43.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:43.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:43.021 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:14:43.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:14:43.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:43.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:43.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:44.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:44.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:44.641 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:14:44.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:14:44.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:44.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:45.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:46.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:46.060 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:14:46.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:46.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:14:46.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:46.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:46.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:47.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:47.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:47.403 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:14:47.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:14:47.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:47.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:48.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:49.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:49.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:49.030 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:14:49.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:14:49.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:49.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:49.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:50.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:50.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:50.761 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:14:50.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:14:50.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:50.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:51.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:52.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:52.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:52.268 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:14:52.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:14:52.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:52.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:52.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:14:53.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:14:53.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:14:53.811 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:14:53.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:14:53.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:14:53.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:14:54.083 INFO:tasks.workunit.client.0.smithi188.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2379663904,v1:127.0.0.1:6851/2379663904] [v2:127.0.0.1:6852/2379663904,v1:127.0.0.1:6853/2379663904] exists,up 156097c6-eff2-4464-a95a-b092a571d27e 2024-02-05T05:14:54.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:14:54.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:14:54.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:14:54.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:14:54.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-02-05T05:14:54.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:14:54.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:14:54.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da5c6933-5054-499f-af1d-dc23404124e5 2024-02-05T05:14:54.090 INFO:tasks.workunit.client.0.smithi188.stdout:add osd7 da5c6933-5054-499f-af1d-dc23404124e5 2024-02-05T05:14:54.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 da5c6933-5054-499f-af1d-dc23404124e5' 2024-02-05T05:14:54.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:14:54.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDObsBlknQkBhAAyZNdLunFQLOpgK2wSMJanQ== 2024-02-05T05:14:54.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDObsBlknQkBhAAyZNdLunFQLOpgK2wSMJanQ=="}' 2024-02-05T05:14:54.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da5c6933-5054-499f-af1d-dc23404124e5 -i td/osd-scrub-repair/7/new.json 2024-02-05T05:14:54.587 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:14:54.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-02-05T05:14:54.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDObsBlknQkBhAAyZNdLunFQLOpgK2wSMJanQ== --osd-uuid da5c6933-5054-499f-af1d-dc23404124e5 2024-02-05T05:14:54.626 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:54.625+0000 7f64f98c54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:54.638 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:54.636+0000 7f64f98c54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:54.639 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:54.637+0000 7f64f98c54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:14:54.639 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:54.638+0000 7f64f98c54c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:14:54.640 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:54.638+0000 7f64f98c54c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:14:54.640 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:54.638+0000 7f64f98c54c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:14:54.730 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:14:54.728+0000 7f64f98c54c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-02-05T05:15:00.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-02-05T05:15:00.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:15:00.612 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd7 key to auth repository 2024-02-05T05:15:00.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-02-05T05:15:00.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:15:00.958 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.7 2024-02-05T05:15:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-02-05T05:15:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:15:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:15:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:15:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:15:00.984 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:00.982+0000 7fbf23c144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:00.986 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:00.984+0000 7fbf23c144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:00.987 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:00.985+0000 7fbf23c144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:01.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-02-05T05:15:01.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:15:01.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-02-05T05:15:01.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:15:01.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:15:01.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:01.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:15:01.227 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:15:01.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:01.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:01.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:01.509 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:01.507+0000 7fbf23c144c0 -1 Falling back to public interface 2024-02-05T05:15:02.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:02.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:02.495 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:15:02.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:15:02.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:02.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:02.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:03.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:03.826 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:15:03.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:03.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:15:03.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:03.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:03.829 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:03.827+0000 7fbf23c144c0 -1 osd.7 0 log_to_monitors true 2024-02-05T05:15:04.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:05.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:05.139 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:15:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:15:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:05.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:05.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:06.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:06.474 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:15:06.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:06.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:15:06.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:06.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:06.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:07.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:07.823 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:15:07.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:07.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:15:07.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:07.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:08.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:09.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:09.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:09.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:15:09.105 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:15:09.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:09.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:10.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:10.470 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:15:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:15:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:10.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:11.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:11.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:11.820 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:15:11.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:15:11.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:11.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:12.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:13.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:13.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:13.098 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:15:13.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:15:13.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:13.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:13.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:14.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:14.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:14.374 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:15:14.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:15:14.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:14.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:14.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:15.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:15.715 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:15:15.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:15.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:15:15.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:15.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:16.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:17.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:17.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:17.266 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:15:17.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:15:17.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:17.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:18.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:19.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:19.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:19.184 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:15:19.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:15:19.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:19.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:19.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:20.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:20.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:20.855 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:15:20.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:15:20.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:20.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:21.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:22.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:22.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:22.825 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:15:22.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:15:22.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:22.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:23.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:24.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:24.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:24.800 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:15:24.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:15:24.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:24.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:25.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:26.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:26.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:26.515 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:15:26.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:15:26.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:26.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:15:26.928 INFO:tasks.workunit.client.0.smithi188.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/4240627395,v1:127.0.0.1:6859/4240627395] [v2:127.0.0.1:6860/4240627395,v1:127.0.0.1:6861/4240627395] exists,up da5c6933-5054-499f-af1d-dc23404124e5 2024-02-05T05:15:26.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:15:26.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:15:26.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:15:26.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:15:26.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:15:26.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c2ad8ae-2d10-44b5-9724-ce075e72a2e6 2024-02-05T05:15:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 8c2ad8ae-2d10-44b5-9724-ce075e72a2e6' 2024-02-05T05:15:26.936 INFO:tasks.workunit.client.0.smithi188.stdout:add osd8 8c2ad8ae-2d10-44b5-9724-ce075e72a2e6 2024-02-05T05:15:26.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:15:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDubsBlpfqFOBAABJyT9eGZSv1CYSDWGGQA/g== 2024-02-05T05:15:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDubsBlpfqFOBAABJyT9eGZSv1CYSDWGGQA/g=="}' 2024-02-05T05:15:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c2ad8ae-2d10-44b5-9724-ce075e72a2e6 -i td/osd-scrub-repair/8/new.json 2024-02-05T05:15:27.311 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:15:27.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-02-05T05:15:27.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDubsBlpfqFOBAABJyT9eGZSv1CYSDWGGQA/g== --osd-uuid 8c2ad8ae-2d10-44b5-9724-ce075e72a2e6 2024-02-05T05:15:27.350 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:27.349+0000 7f38878144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:27.362 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:27.361+0000 7f38878144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:27.363 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:27.362+0000 7f38878144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:27.364 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:27.362+0000 7f38878144c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:15:27.364 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:27.362+0000 7f38878144c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:15:27.364 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:27.362+0000 7f38878144c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:15:27.454 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:27.453+0000 7f38878144c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-02-05T05:15:33.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-02-05T05:15:33.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:15:33.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-02-05T05:15:33.820 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd8 key to auth repository 2024-02-05T05:15:33.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:15:34.164 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.8 2024-02-05T05:15:34.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-02-05T05:15:34.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:15:34.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:15:34.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:15:34.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:15:34.191 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:34.189+0000 7f2776a8c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:34.192 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:34.191+0000 7f2776a8c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:34.193 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:34.192+0000 7f2776a8c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:15:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-02-05T05:15:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:15:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-02-05T05:15:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:15:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:15:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:34.435 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:15:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:15:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:34.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:34.734+0000 7f2776a8c4c0 -1 Falling back to public interface 2024-02-05T05:15:35.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:35.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:35.709 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:15:35.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:15:35.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:35.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:35.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:36.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:36.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:36.987 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:15:36.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:15:36.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:36.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:37.187 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:15:37.185+0000 7f2776a8c4c0 -1 osd.8 0 log_to_monitors true 2024-02-05T05:15:37.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:38.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:38.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:38.279 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:15:38.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:15:38.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:38.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:38.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:39.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:39.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:39.560 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:15:39.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:15:39.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:39.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:39.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:40.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:40.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:40.932 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:15:40.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:15:40.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:40.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:41.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:42.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:42.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:42.211 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:15:42.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:15:42.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:42.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:42.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:43.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:43.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:43.488 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:15:43.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:15:43.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:43.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:43.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:44.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:44.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:44.769 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:15:44.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:15:44.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:44.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:45.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:46.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:46.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:46.161 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:15:46.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:15:46.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:46.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:46.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:47.440 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:15:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:15:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:47.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:47.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:48.750 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:15:48.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:15:48.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:48.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:49.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:50.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:50.078 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:15:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:15:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:50.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:51.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:51.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:51.645 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:15:51.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:15:51.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:51.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:52.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:53.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:53.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:53.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:15:53.297 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:15:53.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:53.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:54.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:55.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:55.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:55.066 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:15:55.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:15:55.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:55.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:55.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:56.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:56.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:56.385 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:15:56.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:15:56.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:56.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:56.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:57.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:57.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:57.872 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:15:57.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:15:57.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:57.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:58.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:15:59.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:15:59.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:15:59.166 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:15:59.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:15:59.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:15:59.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:15:59.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:00.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:00.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:00.912 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:16:00.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:16:00.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:00.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:16:01.195 INFO:tasks.workunit.client.0.smithi188.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/406573001,v1:127.0.0.1:6867/406573001] [v2:127.0.0.1:6868/406573001,v1:127.0.0.1:6869/406573001] exists,up 8c2ad8ae-2d10-44b5-9724-ce075e72a2e6 2024-02-05T05:16:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:16:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:16:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:16:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:16:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-02-05T05:16:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:16:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:16:01.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:16:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 2ff15301-29c5-4bee-8cc3-e4164070278a' 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stdout:add osd9 2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:16:01.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:16:01.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARb8Bl863RDBAARfOlqeGFfPLTWJjifPw+rg== 2024-02-05T05:16:01.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARb8Bl863RDBAARfOlqeGFfPLTWJjifPw+rg=="}' 2024-02-05T05:16:01.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ff15301-29c5-4bee-8cc3-e4164070278a -i td/osd-scrub-repair/9/new.json 2024-02-05T05:16:01.599 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:16:01.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-02-05T05:16:01.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARb8Bl863RDBAARfOlqeGFfPLTWJjifPw+rg== --osd-uuid 2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:16:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:01.637+0000 7f596ac164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:01.692 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:01.690+0000 7f596ac164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:01.692 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:01.691+0000 7f596ac164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:01.693 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:01.692+0000 7f596ac164c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:16:01.693 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:01.692+0000 7f596ac164c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:16:01.693 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:01.692+0000 7f596ac164c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:16:01.792 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:01.790+0000 7f596ac164c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-02-05T05:16:08.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-02-05T05:16:08.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:16:08.079 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd9 key to auth repository 2024-02-05T05:16:08.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-02-05T05:16:08.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:16:08.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-02-05T05:16:08.428 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.9 2024-02-05T05:16:08.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:16:08.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:16:08.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:16:08.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:16:08.454 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:08.453+0000 7f91e2e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:08.455+0000 7f91e2e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:08.457 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:08.456+0000 7f91e2e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:16:08.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-02-05T05:16:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:16:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:16:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:16:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:16:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:16:08.699 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:16:08.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:08.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:08.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:08.988 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:08.987+0000 7f91e2e164c0 -1 Falling back to public interface 2024-02-05T05:16:09.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:09.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:09.965 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:16:09.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:16:09.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:09.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:10.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:11.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:11.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:11.241 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:16:11.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:16:11.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:11.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:11.356 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:16:11.354+0000 7f91e2e164c0 -1 osd.9 0 log_to_monitors true 2024-02-05T05:16:11.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:12.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:12.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:12.557 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:16:12.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:16:12.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:12.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:13.874 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:16:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:16:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:14.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:15.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:15.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:15.156 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:16:15.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:16:15.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:15.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:15.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:16.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:16.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:16.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:16:16.457 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:16:16.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:16.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:16.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:17.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:17.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:17.739 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:16:17.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:16:17.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:17.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:18.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:19.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:19.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:19.018 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:16:19.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:16:19.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:19.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:19.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:20.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:20.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:20.368 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:16:20.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:16:20.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:20.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:20.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:21.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:21.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:21.657 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:16:21.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:16:21.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:21.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:22.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:22.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:22.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:16:22.935 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:16:22.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:22.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:23.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:24.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:24.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:24.303 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:16:24.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:16:24.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:24.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:24.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:25.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:25.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:25.629 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:16:25.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:16:25.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:25.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:25.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:26.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:26.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:26.946 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:16:26.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:16:26.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:26.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:27.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:28.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:28.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:28.737 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:16:28.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:16:28.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:28.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:29.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:30.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:30.360 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:16:30.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:30.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:16:30.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:30.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:30.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:31.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:31.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:31.832 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:16:31.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:16:31.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:31.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:32.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:16:33.506 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:16:33.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:33.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:34.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:35.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:35.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:35.042 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:16:35.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:16:35.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:35.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:35.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:16:36.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:16:36.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:16:36.421 INFO:tasks.workunit.client.0.smithi188.stdout:20 2024-02-05T05:16:36.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:16:36.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:16:36.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:16:36.730 INFO:tasks.workunit.client.0.smithi188.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2239589543,v1:127.0.0.1:6875/2239589543] [v2:127.0.0.1:6876/2239589543,v1:127.0.0.1:6877/2239589543] exists,up 2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:16:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:16:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:16:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:16:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-02-05T05:16:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:16:36.993 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:16:37.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:16:37.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:16:37.416 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:16:37.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:16:38.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:16:41.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-02-05T05:16:41.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:16:41.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:16:41.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:16:41.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:16:41.461 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:16:41.461 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:16:41.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:16:41.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:16:41.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:16:41.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:16:41.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:16:41.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:16:41.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:16:41.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:16:41.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:16:41.970 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:16:41.971 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:16:41.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:16:41.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:41.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:16:42.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-02-05T05:16:42.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-02-05T05:16:42.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-02-05T05:16:42.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:16:42.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673018 2024-02-05T05:16:42.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673018 2024-02-05T05:16:42.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018' 2024-02-05T05:16:42.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:16:42.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509494 2024-02-05T05:16:42.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509494 2024-02-05T05:16:42.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494' 2024-02-05T05:16:42.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.277 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:16:42.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345974 2024-02-05T05:16:42.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345974 2024-02-05T05:16:42.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494 3-85899345974' 2024-02-05T05:16:42.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:16:42.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182439 2024-02-05T05:16:42.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182439 2024-02-05T05:16:42.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494 3-85899345974 4-107374182439' 2024-02-05T05:16:42.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:16:42.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018916 2024-02-05T05:16:42.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018916 2024-02-05T05:16:42.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494 3-85899345974 4-107374182439 5-128849018916' 2024-02-05T05:16:42.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:16:42.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855387 2024-02-05T05:16:42.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855387 2024-02-05T05:16:42.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494 3-85899345974 4-107374182439 5-128849018916 6-150323855387' 2024-02-05T05:16:42.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:16:42.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691859 2024-02-05T05:16:42.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691859 2024-02-05T05:16:42.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494 3-85899345974 4-107374182439 5-128849018916 6-150323855387 7-171798691859' 2024-02-05T05:16:42.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:16:42.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528337 2024-02-05T05:16:42.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528337 2024-02-05T05:16:42.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494 3-85899345974 4-107374182439 5-128849018916 6-150323855387 7-171798691859 8-193273528337' 2024-02-05T05:16:42.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:42.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:16:43.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364805 2024-02-05T05:16:43.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364805 2024-02-05T05:16:43.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673018 2-64424509494 3-85899345974 4-107374182439 5-128849018916 6-150323855387 7-171798691859 8-193273528337 9-214748364805' 2024-02-05T05:16:43.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:43.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-02-05T05:16:43.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:43.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:16:43.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-02-05T05:16:43.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:43.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-02-05T05:16:43.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-02-05T05:16:43.008 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836548 2024-02-05T05:16:43.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:43.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836548 2024-02-05T05:16:43.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:43.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673018 2024-02-05T05:16:43.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:43.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:16:43.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673018 2024-02-05T05:16:43.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:43.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673018 2024-02-05T05:16:43.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673018' 2024-02-05T05:16:43.277 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673018 2024-02-05T05:16:43.277 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:16:43.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673018 -lt 42949673018 2024-02-05T05:16:43.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:43.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509494 2024-02-05T05:16:43.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:43.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:16:43.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509494 2024-02-05T05:16:43.547 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:43.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509494 2024-02-05T05:16:43.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-02-05T05:16:43.548 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509494 2024-02-05T05:16:43.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:16:43.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509494 -lt 64424509494 2024-02-05T05:16:43.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:43.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345974 2024-02-05T05:16:43.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:43.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:16:43.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345974 2024-02-05T05:16:43.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:43.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345974 2024-02-05T05:16:43.820 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345974 2024-02-05T05:16:43.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345974' 2024-02-05T05:16:43.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:16:44.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345974 -lt 85899345974 2024-02-05T05:16:44.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:44.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182439 2024-02-05T05:16:44.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:44.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:16:44.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182439 2024-02-05T05:16:44.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:44.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182439 2024-02-05T05:16:44.090 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.4 seq 107374182439 2024-02-05T05:16:44.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2024-02-05T05:16:44.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:16:44.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182439 -lt 107374182439 2024-02-05T05:16:44.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:44.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018916 2024-02-05T05:16:44.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:44.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:16:44.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018916 2024-02-05T05:16:44.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:44.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018916 2024-02-05T05:16:44.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018916' 2024-02-05T05:16:44.362 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.5 seq 128849018916 2024-02-05T05:16:44.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:16:44.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018915 -lt 128849018916 2024-02-05T05:16:44.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:16:45.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:16:45.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:16:45.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018917 -lt 128849018916 2024-02-05T05:16:45.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:45.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855387 2024-02-05T05:16:45.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:45.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:16:45.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855387 2024-02-05T05:16:45.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:45.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855387 2024-02-05T05:16:45.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855387' 2024-02-05T05:16:45.932 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.6 seq 150323855387 2024-02-05T05:16:45.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:16:46.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855388 -lt 150323855387 2024-02-05T05:16:46.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:46.199 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691859 2024-02-05T05:16:46.199 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:46.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:16:46.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691859 2024-02-05T05:16:46.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:46.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691859 2024-02-05T05:16:46.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691859' 2024-02-05T05:16:46.202 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.7 seq 171798691859 2024-02-05T05:16:46.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:16:46.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691859 -lt 171798691859 2024-02-05T05:16:46.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:46.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528337 2024-02-05T05:16:46.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:46.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:16:46.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528337 2024-02-05T05:16:46.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:46.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528337 2024-02-05T05:16:46.475 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.8 seq 193273528337 2024-02-05T05:16:46.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528337' 2024-02-05T05:16:46.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:16:46.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528337 -lt 193273528337 2024-02-05T05:16:46.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:46.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364805 2024-02-05T05:16:46.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:46.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:16:46.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364805 2024-02-05T05:16:46.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:46.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364805 2024-02-05T05:16:46.748 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.9 seq 214748364805 2024-02-05T05:16:46.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364805' 2024-02-05T05:16:46.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:16:47.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364805 2024-02-05T05:16:47.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:16:47.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:47.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:47.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:16:47.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:16:47.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:16:47.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:16:47.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:16:47.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:16:47.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:16:47.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:16:47.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:16:47.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:16:47.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:47.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:16:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:16:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-02-05T05:16:48.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:16:48.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:16:49.003 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:16:49.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:16:50.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:16:50.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:16:50.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:16:50.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:16:50.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:16:50.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:16:50.018 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:16:50.018 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:16:50.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:16:50.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:16:50.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:16:50.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:16:50.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:16:50.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:16:50.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:16:50.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:16:50.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:16:50.444 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:16:50.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:16:50.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:50.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:16:50.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-02-05T05:16:50.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-02-05T05:16:50.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-02-05T05:16:50.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:50.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:16:50.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673021 2024-02-05T05:16:50.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673021 2024-02-05T05:16:50.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021' 2024-02-05T05:16:50.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:50.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:16:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509497 2024-02-05T05:16:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509497 2024-02-05T05:16:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497' 2024-02-05T05:16:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:51.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:16:51.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345976 2024-02-05T05:16:51.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345976 2024-02-05T05:16:51.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497 3-85899345976' 2024-02-05T05:16:51.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:51.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:16:51.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182442 2024-02-05T05:16:51.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182442 2024-02-05T05:16:51.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497 3-85899345976 4-107374182442' 2024-02-05T05:16:51.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:51.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:16:51.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018919 2024-02-05T05:16:51.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018919 2024-02-05T05:16:51.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497 3-85899345976 4-107374182442 5-128849018919' 2024-02-05T05:16:51.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:51.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:16:51.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-02-05T05:16:51.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-02-05T05:16:51.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497 3-85899345976 4-107374182442 5-128849018919 6-150323855390' 2024-02-05T05:16:51.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:51.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:16:51.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691861 2024-02-05T05:16:51.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691861 2024-02-05T05:16:51.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497 3-85899345976 4-107374182442 5-128849018919 6-150323855390 7-171798691861' 2024-02-05T05:16:51.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:51.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:16:51.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528340 2024-02-05T05:16:51.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528340 2024-02-05T05:16:51.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497 3-85899345976 4-107374182442 5-128849018919 6-150323855390 7-171798691861 8-193273528340' 2024-02-05T05:16:51.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:16:51.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:16:51.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364808 2024-02-05T05:16:51.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364808 2024-02-05T05:16:51.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673021 2-64424509497 3-85899345976 4-107374182442 5-128849018919 6-150323855390 7-171798691861 8-193273528340 9-214748364808' 2024-02-05T05:16:51.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:51.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-02-05T05:16:51.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:51.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:16:51.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-02-05T05:16:51.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:51.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-02-05T05:16:51.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-02-05T05:16:51.958 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836550 2024-02-05T05:16:51.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:52.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836550 2024-02-05T05:16:52.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:16:53.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:16:53.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:16:53.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836551 -lt 21474836550 2024-02-05T05:16:53.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:53.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673021 2024-02-05T05:16:53.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:53.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:16:53.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673021 2024-02-05T05:16:53.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:53.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673021 2024-02-05T05:16:53.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673021' 2024-02-05T05:16:53.507 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673021 2024-02-05T05:16:53.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:16:53.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673021 -lt 42949673021 2024-02-05T05:16:53.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:53.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509497 2024-02-05T05:16:53.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:53.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:16:53.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509497 2024-02-05T05:16:53.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:53.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509497 2024-02-05T05:16:53.779 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509497 2024-02-05T05:16:53.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2024-02-05T05:16:53.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:16:54.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509497 -lt 64424509497 2024-02-05T05:16:54.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:54.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345976 2024-02-05T05:16:54.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:54.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:16:54.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345976 2024-02-05T05:16:54.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:54.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345976 2024-02-05T05:16:54.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345976' 2024-02-05T05:16:54.052 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345976 2024-02-05T05:16:54.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:16:54.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345977 -lt 85899345976 2024-02-05T05:16:54.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:54.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182442 2024-02-05T05:16:54.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:54.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:16:54.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182442 2024-02-05T05:16:54.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:54.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182442 2024-02-05T05:16:54.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182442' 2024-02-05T05:16:54.326 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.4 seq 107374182442 2024-02-05T05:16:54.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:16:54.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182442 -lt 107374182442 2024-02-05T05:16:54.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:54.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018919 2024-02-05T05:16:54.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:54.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:16:54.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018919 2024-02-05T05:16:54.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:54.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018919 2024-02-05T05:16:54.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018919' 2024-02-05T05:16:54.629 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.5 seq 128849018919 2024-02-05T05:16:54.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:16:54.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018920 -lt 128849018919 2024-02-05T05:16:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:54.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-02-05T05:16:54.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:54.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:16:54.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-02-05T05:16:54.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:54.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-02-05T05:16:54.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-02-05T05:16:54.922 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.6 seq 150323855390 2024-02-05T05:16:54.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:16:55.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855391 -lt 150323855390 2024-02-05T05:16:55.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:55.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691861 2024-02-05T05:16:55.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:55.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:16:55.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691861 2024-02-05T05:16:55.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:55.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691861 2024-02-05T05:16:55.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-02-05T05:16:55.190 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.7 seq 171798691861 2024-02-05T05:16:55.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:16:55.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691861 2024-02-05T05:16:55.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:55.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528340 2024-02-05T05:16:55.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:55.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:16:55.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528340 2024-02-05T05:16:55.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:55.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528340 2024-02-05T05:16:55.459 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.8 seq 193273528340 2024-02-05T05:16:55.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2024-02-05T05:16:55.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:16:55.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528340 -lt 193273528340 2024-02-05T05:16:55.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:16:55.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364808 2024-02-05T05:16:55.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:16:55.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:16:55.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364808 2024-02-05T05:16:55.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:16:55.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364808 2024-02-05T05:16:55.730 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.9 seq 214748364808 2024-02-05T05:16:55.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364808' 2024-02-05T05:16:55.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:16:55.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364808 -lt 214748364808 2024-02-05T05:16:55.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:16:55.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:55.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:56.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:16:56.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:16:56.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:16:56.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:16:56.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:16:56.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:16:56.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:16:56.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:16:56.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:16:56.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:16:56.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:16:56.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:16:56.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:16:56.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:16:56.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:16:56.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:16:56.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:16:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:16:57.303 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:16:57.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:16:57.874 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:16:57.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:16:57.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:16:57.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:16:58.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:16:58.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:16:58.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:16:58.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:16:58.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:16:58.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:16:58.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:16:58.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:16:58.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:16:58.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:16:58.669 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:16:58.669 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr:9 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr:7' 2024-02-05T05:16:58.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:16:58.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:16:58.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:16:59.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:16:59.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:16:59.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:16:59.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:16:59.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:16:59.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:16:59.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:16:59.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:17:01.156 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:17:01.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:17:01.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:01.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:17:01.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:17:01.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:17:01.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:17:01.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:17:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:17:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:17:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:17:01.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:17:01.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:17:01.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:17:01.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:17:01.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:17:01.713 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:01.711+0000 7f78ae7074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:01.715 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:01.713+0000 7f78ae7074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:01.716 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:01.715+0000 7f78ae7074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:01.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:17:01.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:01.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:17:01.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:01.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:01.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:01.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:01.962 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:17:01.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:01.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:17:02.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:02.247 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:02.246+0000 7f78ae7074c0 -1 Falling back to public interface 2024-02-05T05:17:03.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:03.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:03.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:03.231 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:03.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:03.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:17:03.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:04.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:04.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:04.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:04.510 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:04.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:04.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:17:04.862 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:04.861+0000 7f78ae7074c0 -1 osd.3 67 log_to_monitors true 2024-02-05T05:17:04.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:05.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:05.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:05.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:05.876 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:05.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:05.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:17:06.294 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 71 up_thru 71 down_at 68 last_clean_interval [20,67) [v2:127.0.0.1:6826/413561040,v1:127.0.0.1:6827/413561040] [v2:127.0.0.1:6828/413561040,v1:127.0.0.1:6829/413561040] exists,up 3e670ed6-5780-4b95-8c34-207ce6eba772 2024-02-05T05:17:06.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:06.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:06.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:17:06.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:17:06.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:17:06.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:17:06.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:17:06.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:17:06.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:17:06.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:17:06.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:17:06.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:17:06.760 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:17:06.761 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:17:06.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:17:06.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:06.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:17:06.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836555 2024-02-05T05:17:06.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836555 2024-02-05T05:17:06.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555' 2024-02-05T05:17:06.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:06.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:17:07.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-02-05T05:17:07.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-02-05T05:17:07.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025' 2024-02-05T05:17:07.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:17:07.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-02-05T05:17:07.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-02-05T05:17:07.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501' 2024-02-05T05:17:07.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:17:07.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-02-05T05:17:07.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-02-05T05:17:07.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501 3-304942678018' 2024-02-05T05:17:07.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:17:07.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182446 2024-02-05T05:17:07.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182446 2024-02-05T05:17:07.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501 3-304942678018 4-107374182446' 2024-02-05T05:17:07.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:17:07.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018923 2024-02-05T05:17:07.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018923 2024-02-05T05:17:07.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501 3-304942678018 4-107374182446 5-128849018923' 2024-02-05T05:17:07.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.549 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:17:07.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855394 2024-02-05T05:17:07.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855394 2024-02-05T05:17:07.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501 3-304942678018 4-107374182446 5-128849018923 6-150323855394' 2024-02-05T05:17:07.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:17:07.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691866 2024-02-05T05:17:07.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691866 2024-02-05T05:17:07.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501 3-304942678018 4-107374182446 5-128849018923 6-150323855394 7-171798691866' 2024-02-05T05:17:07.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.754 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:17:07.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528344 2024-02-05T05:17:07.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528344 2024-02-05T05:17:07.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501 3-304942678018 4-107374182446 5-128849018923 6-150323855394 7-171798691866 8-193273528344' 2024-02-05T05:17:07.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:07.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:17:07.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364812 2024-02-05T05:17:07.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364812 2024-02-05T05:17:07.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673025 2-64424509501 3-304942678018 4-107374182446 5-128849018923 6-150323855394 7-171798691866 8-193273528344 9-214748364812' 2024-02-05T05:17:07.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:07.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836555 2024-02-05T05:17:07.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:07.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:17:07.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836555 2024-02-05T05:17:07.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:07.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836555 2024-02-05T05:17:07.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-02-05T05:17:07.964 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836555 2024-02-05T05:17:07.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836555 2024-02-05T05:17:08.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:17:09.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:17:09.233 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:09.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836555 2024-02-05T05:17:09.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:09.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-02-05T05:17:09.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:09.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:17:09.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-02-05T05:17:09.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:09.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-02-05T05:17:09.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-02-05T05:17:09.513 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673025 2024-02-05T05:17:09.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:17:09.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673025 2024-02-05T05:17:09.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:09.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-02-05T05:17:09.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:09.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:17:09.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-02-05T05:17:09.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:09.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-02-05T05:17:09.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-02-05T05:17:09.784 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509501 2024-02-05T05:17:09.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:17:10.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509501 2024-02-05T05:17:10.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:10.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678018 2024-02-05T05:17:10.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:10.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:17:10.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678018 2024-02-05T05:17:10.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:10.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-02-05T05:17:10.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678018' 2024-02-05T05:17:10.055 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 304942678018 2024-02-05T05:17:10.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:17:10.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-02-05T05:17:10.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:10.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182446 2024-02-05T05:17:10.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:10.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:17:10.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182446 2024-02-05T05:17:10.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:10.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182446 2024-02-05T05:17:10.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182446' 2024-02-05T05:17:10.327 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182446 2024-02-05T05:17:10.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:17:10.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182446 -lt 107374182446 2024-02-05T05:17:10.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:10.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018923 2024-02-05T05:17:10.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:10.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:17:10.599 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018923 2024-02-05T05:17:10.599 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:10.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018923 2024-02-05T05:17:10.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-02-05T05:17:10.600 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 128849018923 2024-02-05T05:17:10.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:17:10.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018924 -lt 128849018923 2024-02-05T05:17:10.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:10.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855394 2024-02-05T05:17:10.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:10.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:17:10.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855394 2024-02-05T05:17:10.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:10.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855394 2024-02-05T05:17:10.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2024-02-05T05:17:10.911 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855394 2024-02-05T05:17:10.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:17:11.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855395 -lt 150323855394 2024-02-05T05:17:11.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:11.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691866 2024-02-05T05:17:11.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:17:11.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691866 2024-02-05T05:17:11.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:11.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691866 2024-02-05T05:17:11.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-02-05T05:17:11.187 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691866 2024-02-05T05:17:11.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:17:11.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691866 -lt 171798691866 2024-02-05T05:17:11.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:11.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528344 2024-02-05T05:17:11.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:11.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:17:11.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528344 2024-02-05T05:17:11.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:11.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528344 2024-02-05T05:17:11.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2024-02-05T05:17:11.460 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528344 2024-02-05T05:17:11.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:17:11.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528344 -lt 193273528344 2024-02-05T05:17:11.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:11.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364812 2024-02-05T05:17:11.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:11.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:17:11.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364812 2024-02-05T05:17:11.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:11.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364812 2024-02-05T05:17:11.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364812' 2024-02-05T05:17:11.732 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 214748364812 2024-02-05T05:17:11.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:17:12.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364812 -lt 214748364812 2024-02-05T05:17:12.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:17:12.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:12.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:12.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:17:12.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:17:12.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:17:12.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:17:12.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:17:12.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:17:12.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:17:12.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:17:12.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:17:12.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:17:12.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:12.599 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:12.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:17:12.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:17:12.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:17:12.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:17:12.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:17:12.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:17:12.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:17:12.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:17:13.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:17:13.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:17:13.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:17:13.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:17:13.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:13.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:13.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:13.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:13.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:16:48.892914+0000 2024-02-05T05:17:13.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:17:13.724 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:17:13.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:16:48.892914+0000 2024-02-05T05:17:13.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:17:13.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:16:48.892914+0000 2024-02-05T05:17:13.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:17:13.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:17:13.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:13.738 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:13.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:13.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:13.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:13.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:14.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:16:48.892914+0000 '>' 2024-02-05T05:16:48.892914+0000 2024-02-05T05:17:14.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:17:15.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:17:15.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:15.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:15.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:15.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:15.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:15.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:16:48.892914+0000 '>' 2024-02-05T05:16:48.892914+0000 2024-02-05T05:17:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:17:16.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:17:16.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:16.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:16.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:16.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:16.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:16.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:16.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:13.869181+0000 '>' 2024-02-05T05:16:48.892914+0000 2024-02-05T05:17:16.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:17:16.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:17:16.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:17:16.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:17:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:17:17.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:17:17.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:17:17.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:17:17.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:17:17.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:17:17.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:17:17.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:17:17.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:17:19.316 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:17:19.316 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:17:19.316 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:17:19.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:17:19.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:19.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:17:19.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:17:19.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:17:19.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:17:19.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:19.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:19.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:19.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:19.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:17:19.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:17:19.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:19.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:17:19.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:19.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:17:19.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:19.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.9 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:17:19.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:17:19.619 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:19.616+0000 7f4b6140e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:19.620 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:19.619+0000 7f4b6140e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:19.621 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:19.620+0000 7f4b6140e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:17:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:17:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:17:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:19.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:20.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:20.143 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:20.142+0000 7f4b6140e4c0 -1 Falling back to public interface 2024-02-05T05:17:21.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:21.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:21.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:21.139 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:21.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:21.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:21.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:22.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:22.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:22.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:22.418 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:22.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:22.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:22.452 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:22.451+0000 7f4b6140e4c0 -1 osd.9 72 log_to_monitors true 2024-02-05T05:17:22.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:23.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:23.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:23.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:23.701 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:23.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:23.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:23.979 INFO:tasks.workunit.client.0.smithi188.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 73 last_clean_interval [50,72) [v2:127.0.0.1:6874/1154237379,v1:127.0.0.1:6875/1154237379] [v2:127.0.0.1:6876/1154237379,v1:127.0.0.1:6877/1154237379] exists,up 2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:17:23.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:23.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:23.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:23.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:17:23.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:17:23.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:17:24.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:17:24.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:17:24.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:17:24.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:17:24.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:17:24.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:17:24.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:24.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:17:24.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836559 2024-02-05T05:17:24.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836559 2024-02-05T05:17:24.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559' 2024-02-05T05:17:24.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:24.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:17:25.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673030 2024-02-05T05:17:25.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673030 2024-02-05T05:17:25.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030' 2024-02-05T05:17:25.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.043 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:17:25.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-02-05T05:17:25.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-02-05T05:17:25.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506' 2024-02-05T05:17:25.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:17:25.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678023 2024-02-05T05:17:25.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678023 2024-02-05T05:17:25.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506 3-304942678023' 2024-02-05T05:17:25.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.286 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:17:25.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182450 2024-02-05T05:17:25.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182450 2024-02-05T05:17:25.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506 3-304942678023 4-107374182450' 2024-02-05T05:17:25.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:17:25.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018928 2024-02-05T05:17:25.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018928 2024-02-05T05:17:25.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506 3-304942678023 4-107374182450 5-128849018928' 2024-02-05T05:17:25.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:17:25.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855399 2024-02-05T05:17:25.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855399 2024-02-05T05:17:25.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506 3-304942678023 4-107374182450 5-128849018928 6-150323855399' 2024-02-05T05:17:25.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:17:25.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691870 2024-02-05T05:17:25.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691870 2024-02-05T05:17:25.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506 3-304942678023 4-107374182450 5-128849018928 6-150323855399 7-171798691870' 2024-02-05T05:17:25.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:17:25.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528348 2024-02-05T05:17:25.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528348 2024-02-05T05:17:25.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506 3-304942678023 4-107374182450 5-128849018928 6-150323855399 7-171798691870 8-193273528348' 2024-02-05T05:17:25.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:25.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:17:25.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-02-05T05:17:25.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-02-05T05:17:25.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509506 3-304942678023 4-107374182450 5-128849018928 6-150323855399 7-171798691870 8-193273528348 9-330712481794' 2024-02-05T05:17:25.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:25.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836559 2024-02-05T05:17:25.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:25.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:17:25.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836559 2024-02-05T05:17:25.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:25.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836559 2024-02-05T05:17:25.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-02-05T05:17:25.903 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836559 2024-02-05T05:17:25.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:26.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836559 2024-02-05T05:17:26.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:17:27.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:17:27.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:27.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836559 2024-02-05T05:17:27.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:27.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673030 2024-02-05T05:17:27.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:27.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:17:27.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673030 2024-02-05T05:17:27.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:27.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673030 2024-02-05T05:17:27.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-02-05T05:17:27.452 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673030 2024-02-05T05:17:27.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:17:27.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673030 -lt 42949673030 2024-02-05T05:17:27.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:27.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-02-05T05:17:27.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:27.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:17:27.725 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-02-05T05:17:27.725 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-02-05T05:17:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-02-05T05:17:27.726 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509506 2024-02-05T05:17:27.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:17:27.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509506 -lt 64424509506 2024-02-05T05:17:27.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:27.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678023 2024-02-05T05:17:27.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:27.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:17:28.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678023 2024-02-05T05:17:28.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:28.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678023 2024-02-05T05:17:28.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678023' 2024-02-05T05:17:28.001 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 304942678023 2024-02-05T05:17:28.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:17:28.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678023 -lt 304942678023 2024-02-05T05:17:28.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:28.287 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182450 2024-02-05T05:17:28.287 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:28.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:17:28.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182450 2024-02-05T05:17:28.289 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:28.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182450 2024-02-05T05:17:28.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182450' 2024-02-05T05:17:28.290 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182450 2024-02-05T05:17:28.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:17:28.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182451 -lt 107374182450 2024-02-05T05:17:28.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:28.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018928 2024-02-05T05:17:28.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:28.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:17:28.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018928 2024-02-05T05:17:28.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:28.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018928 2024-02-05T05:17:28.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018928' 2024-02-05T05:17:28.561 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 128849018928 2024-02-05T05:17:28.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:17:28.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018928 -lt 128849018928 2024-02-05T05:17:28.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:28.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855399 2024-02-05T05:17:28.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:28.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:17:28.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855399 2024-02-05T05:17:28.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:28.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855399 2024-02-05T05:17:28.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-02-05T05:17:28.848 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855399 2024-02-05T05:17:28.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:17:29.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855399 2024-02-05T05:17:29.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:29.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691870 2024-02-05T05:17:29.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:29.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:17:29.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691870 2024-02-05T05:17:29.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:29.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691870 2024-02-05T05:17:29.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-02-05T05:17:29.119 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691870 2024-02-05T05:17:29.119 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:17:29.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691871 -lt 171798691870 2024-02-05T05:17:29.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:29.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528348 2024-02-05T05:17:29.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:29.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:17:29.387 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528348 2024-02-05T05:17:29.387 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:29.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528348 2024-02-05T05:17:29.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2024-02-05T05:17:29.388 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528348 2024-02-05T05:17:29.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:17:29.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528349 -lt 193273528348 2024-02-05T05:17:29.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:29.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-02-05T05:17:29.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:29.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:17:29.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-02-05T05:17:29.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:29.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-02-05T05:17:29.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-02-05T05:17:29.661 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 330712481794 2024-02-05T05:17:29.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:17:29.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-02-05T05:17:29.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:17:29.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:29.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:30.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:17:30.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:17:30.262 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:17:30.262 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:17:30.262 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:17:30.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:17:30.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:17:30.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:17:30.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:17:30.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:17:30.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:30.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:30.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:17:30.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:17:30.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:17:30.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:17:30.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:17:30.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:17:30.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:17:30.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:17:30.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:17:30.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:17:30.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:17:30.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:17:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:17:33.259 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:33.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:17:33.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:17:33.778 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.5 2024-02-05T05:17:33.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:17:33.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:17:33.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:17:33.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:17:33.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:17:33.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:17:33.805 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:33.803+0000 7ff4d63074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:33.808 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:33.806+0000 7ff4d63074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:33.809 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:33.807+0000 7ff4d63074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:34.056 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:17:34.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:34.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:17:34.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:34.334 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:34.332+0000 7ff4d63074c0 -1 Falling back to public interface 2024-02-05T05:17:35.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:35.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:35.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:35.325 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:35.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:35.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:17:35.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:36.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:36.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:36.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:36.603 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:36.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:36.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:17:36.803 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:36.801+0000 7ff4d63074c0 -1 osd.5 79 log_to_monitors true 2024-02-05T05:17:36.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:37.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:17:38.404 INFO:tasks.workunit.client.0.smithi188.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/361527229,v1:127.0.0.1:6843/361527229] [v2:127.0.0.1:6844/361527229,v1:127.0.0.1:6845/361527229] exists,up a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:17:38.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:17:38.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:17:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:17:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:17:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:17:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:17:38.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:17:38.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:17:38.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:17:38.787 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:38.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:17:38.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836563 2024-02-05T05:17:38.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836563 2024-02-05T05:17:38.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563' 2024-02-05T05:17:38.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:38.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:17:39.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-02-05T05:17:39.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-02-05T05:17:39.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033' 2024-02-05T05:17:39.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:17:39.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-02-05T05:17:39.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-02-05T05:17:39.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510' 2024-02-05T05:17:39.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:17:39.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678026 2024-02-05T05:17:39.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678026 2024-02-05T05:17:39.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-304942678026' 2024-02-05T05:17:39.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:17:39.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182454 2024-02-05T05:17:39.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182454 2024-02-05T05:17:39.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-304942678026 4-107374182454' 2024-02-05T05:17:39.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:17:39.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-02-05T05:17:39.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-02-05T05:17:39.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-304942678026 4-107374182454 5-356482285570' 2024-02-05T05:17:39.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:17:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855403 2024-02-05T05:17:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855403 2024-02-05T05:17:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-304942678026 4-107374182454 5-356482285570 6-150323855403' 2024-02-05T05:17:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:17:39.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691874 2024-02-05T05:17:39.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691874 2024-02-05T05:17:39.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-304942678026 4-107374182454 5-356482285570 6-150323855403 7-171798691874' 2024-02-05T05:17:39.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:17:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528352 2024-02-05T05:17:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528352 2024-02-05T05:17:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-304942678026 4-107374182454 5-356482285570 6-150323855403 7-171798691874 8-193273528352' 2024-02-05T05:17:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:39.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:17:39.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-02-05T05:17:39.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-02-05T05:17:39.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673033 2-64424509510 3-304942678026 4-107374182454 5-356482285570 6-150323855403 7-171798691874 8-193273528352 9-330712481798' 2024-02-05T05:17:39.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:39.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836563 2024-02-05T05:17:39.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:39.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:17:39.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836563 2024-02-05T05:17:39.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:39.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836563 2024-02-05T05:17:39.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-02-05T05:17:39.954 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836563 2024-02-05T05:17:39.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:40.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836563 2024-02-05T05:17:40.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:17:41.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:17:41.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:17:41.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836563 2024-02-05T05:17:41.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:41.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-02-05T05:17:41.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:41.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:17:41.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-02-05T05:17:41.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:41.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-02-05T05:17:41.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-02-05T05:17:41.500 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673033 2024-02-05T05:17:41.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:17:41.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673034 -lt 42949673033 2024-02-05T05:17:41.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:41.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-02-05T05:17:41.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:41.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:17:41.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-02-05T05:17:41.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:41.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-02-05T05:17:41.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-02-05T05:17:41.772 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509510 2024-02-05T05:17:41.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:17:42.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-02-05T05:17:42.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:42.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678026 2024-02-05T05:17:42.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:42.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:17:42.043 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678026 2024-02-05T05:17:42.043 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678026 2024-02-05T05:17:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678026' 2024-02-05T05:17:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 304942678026 2024-02-05T05:17:42.044 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:17:42.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678027 -lt 304942678026 2024-02-05T05:17:42.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:42.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182454 2024-02-05T05:17:42.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:42.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:17:42.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182454 2024-02-05T05:17:42.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:42.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182454 2024-02-05T05:17:42.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182454' 2024-02-05T05:17:42.314 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182454 2024-02-05T05:17:42.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:17:42.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182454 -lt 107374182454 2024-02-05T05:17:42.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:42.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285570 2024-02-05T05:17:42.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:42.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:17:42.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285570 2024-02-05T05:17:42.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:42.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-02-05T05:17:42.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285570' 2024-02-05T05:17:42.585 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 356482285570 2024-02-05T05:17:42.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:17:42.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285570 2024-02-05T05:17:42.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:42.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855403 2024-02-05T05:17:42.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:42.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:17:42.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855403 2024-02-05T05:17:42.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:42.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855403 2024-02-05T05:17:42.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-02-05T05:17:42.876 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855403 2024-02-05T05:17:42.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:17:43.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855403 -lt 150323855403 2024-02-05T05:17:43.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:43.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691874 2024-02-05T05:17:43.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:43.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:17:43.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691874 2024-02-05T05:17:43.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:43.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691874 2024-02-05T05:17:43.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-02-05T05:17:43.147 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691874 2024-02-05T05:17:43.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:17:43.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691875 -lt 171798691874 2024-02-05T05:17:43.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:43.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528352 2024-02-05T05:17:43.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:43.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:17:43.416 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528352 2024-02-05T05:17:43.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:43.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528352 2024-02-05T05:17:43.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2024-02-05T05:17:43.418 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528352 2024-02-05T05:17:43.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:17:43.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528353 -lt 193273528352 2024-02-05T05:17:43.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:17:43.688 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-02-05T05:17:43.688 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:17:43.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:17:43.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-02-05T05:17:43.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:17:43.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-02-05T05:17:43.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-02-05T05:17:43.691 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 330712481798 2024-02-05T05:17:43.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:17:43.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-02-05T05:17:43.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:17:43.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:43.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:44.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:17:44.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:17:44.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:17:44.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:17:44.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:17:44.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:17:44.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:17:44.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:17:44.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:17:44.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:17:44.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:17:44.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:17:44.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:17:45.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:17:45.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:17:45.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:17:45.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:17:45.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:45.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:45.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:45.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:45.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:45.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:17:45.704 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:17:45.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:45.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:45.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:45.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:13.869181+0000 '>' 2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:45.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:17:46.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:17:46.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:46.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:46.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:46.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:46.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:46.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:47.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:13.869181+0000 '>' 2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:47.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:17:48.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:17:48.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:48.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:48.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:48.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:48.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:48.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:48.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:13.869181+0000 '>' 2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:48.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:17:49.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:17:49.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:49.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:49.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:49.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:49.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:49.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:49.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:13.869181+0000 '>' 2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:49.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:17:50.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:17:50.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:17:50.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:17:50.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:17:50.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:17:50.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:17:50.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:17:51.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:45.873472+0000 '>' 2024-02-05T05:17:13.869181+0000 2024-02-05T05:17:51.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:17:51.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:17:51.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:17:51.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:17:53.301 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:17:53.302 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:17:53.302 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:17:53.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:17:53.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:17:53.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:17:53.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:17:53.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:17:53.570 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:17:53.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.9 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:17:53.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:17:53.599 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:53.597+0000 7f0f5e2114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:53.601 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:53.600+0000 7f0f5e2114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:53.602 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:53.601+0000 7f0f5e2114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:17:53.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:53.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:54.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:54.118 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:54.117+0000 7f0f5e2114c0 -1 Falling back to public interface 2024-02-05T05:17:55.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:55.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:55.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:17:55.114 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:55.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:55.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:55.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:56.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:56.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:56.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:17:56.392 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:56.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:56.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:56.398 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:17:56.396+0000 7f0f5e2114c0 -1 osd.9 84 log_to_monitors true 2024-02-05T05:17:56.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:17:57.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:17:57.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:17:57.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:17:57.784 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:57.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:17:57.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:17:58.075 INFO:tasks.workunit.client.0.smithi188.stderr:osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6874/2912599681,v1:127.0.0.1:6875/2912599681] [v2:127.0.0.1:6876/2912599681,v1:127.0.0.1:6877/2912599681] exists,up 2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:17:58.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:17:58.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:17:58.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:17:58.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:17:58.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:17:58.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:17:58.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:17:58.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:17:58.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:17:58.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:17:58.720 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:17:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:17:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:17:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:17:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:58.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:17:58.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-02-05T05:17:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-02-05T05:17:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-02-05T05:17:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:58.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:17:59.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-02-05T05:17:59.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-02-05T05:17:59.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038' 2024-02-05T05:17:59.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:59.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:17:59.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-02-05T05:17:59.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-02-05T05:17:59.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515' 2024-02-05T05:17:59.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:59.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:17:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678031 2024-02-05T05:17:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678031 2024-02-05T05:17:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515 3-304942678031' 2024-02-05T05:17:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:59.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:17:59.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182459 2024-02-05T05:17:59.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182459 2024-02-05T05:17:59.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515 3-304942678031 4-107374182459' 2024-02-05T05:17:59.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:59.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:17:59.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285575 2024-02-05T05:17:59.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285575 2024-02-05T05:17:59.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515 3-304942678031 4-107374182459 5-356482285575' 2024-02-05T05:17:59.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:59.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:17:59.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855408 2024-02-05T05:17:59.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855408 2024-02-05T05:17:59.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515 3-304942678031 4-107374182459 5-356482285575 6-150323855408' 2024-02-05T05:17:59.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:59.842 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:17:59.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691879 2024-02-05T05:17:59.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691879 2024-02-05T05:17:59.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515 3-304942678031 4-107374182459 5-356482285575 6-150323855408 7-171798691879' 2024-02-05T05:17:59.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:17:59.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:18:00.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528357 2024-02-05T05:18:00.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528357 2024-02-05T05:18:00.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515 3-304942678031 4-107374182459 5-356482285575 6-150323855408 7-171798691879 8-193273528357' 2024-02-05T05:18:00.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:00.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:18:00.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-02-05T05:18:00.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-02-05T05:18:00.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673038 2-64424509515 3-304942678031 4-107374182459 5-356482285575 6-150323855408 7-171798691879 8-193273528357 9-377957122050' 2024-02-05T05:18:00.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:00.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-02-05T05:18:00.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:00.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:00.152 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-02-05T05:18:00.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:00.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-02-05T05:18:00.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-02-05T05:18:00.154 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836568 2024-02-05T05:18:00.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:00.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836568 2024-02-05T05:18:00.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:18:01.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:18:01.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:01.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836568 2024-02-05T05:18:01.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:01.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-02-05T05:18:01.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:01.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:01.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-02-05T05:18:01.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:01.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-02-05T05:18:01.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-02-05T05:18:01.697 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673038 2024-02-05T05:18:01.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:01.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673039 -lt 42949673038 2024-02-05T05:18:01.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:01.965 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-02-05T05:18:01.965 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:01.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:01.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-02-05T05:18:01.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:01.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-02-05T05:18:01.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-02-05T05:18:01.967 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509515 2024-02-05T05:18:01.968 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:02.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509515 2024-02-05T05:18:02.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:02.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678031 2024-02-05T05:18:02.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:02.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:18:02.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678031 2024-02-05T05:18:02.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:02.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678031 2024-02-05T05:18:02.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678031' 2024-02-05T05:18:02.241 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 304942678031 2024-02-05T05:18:02.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:18:02.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678032 -lt 304942678031 2024-02-05T05:18:02.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:02.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182459 2024-02-05T05:18:02.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:02.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:18:02.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182459 2024-02-05T05:18:02.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:02.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182459 2024-02-05T05:18:02.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182459' 2024-02-05T05:18:02.513 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182459 2024-02-05T05:18:02.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:18:02.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182459 2024-02-05T05:18:02.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:02.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285575 2024-02-05T05:18:02.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:02.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:18:02.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285575 2024-02-05T05:18:02.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:02.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285575 2024-02-05T05:18:02.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285575' 2024-02-05T05:18:02.799 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 356482285575 2024-02-05T05:18:02.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:18:03.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285576 -lt 356482285575 2024-02-05T05:18:03.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:03.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855408 2024-02-05T05:18:03.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:03.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:18:03.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855408 2024-02-05T05:18:03.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:03.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855408 2024-02-05T05:18:03.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855408' 2024-02-05T05:18:03.070 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855408 2024-02-05T05:18:03.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:18:03.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855408 -lt 150323855408 2024-02-05T05:18:03.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:03.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691879 2024-02-05T05:18:03.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:03.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:18:03.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691879 2024-02-05T05:18:03.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:03.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691879 2024-02-05T05:18:03.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691879' 2024-02-05T05:18:03.340 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691879 2024-02-05T05:18:03.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:18:03.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691880 -lt 171798691879 2024-02-05T05:18:03.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:03.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528357 2024-02-05T05:18:03.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:03.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:18:03.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528357 2024-02-05T05:18:03.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:03.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528357 2024-02-05T05:18:03.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-02-05T05:18:03.611 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528357 2024-02-05T05:18:03.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:18:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528357 2024-02-05T05:18:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:03.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-377957122050 2024-02-05T05:18:03.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:03.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:18:03.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-377957122050 2024-02-05T05:18:03.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:03.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-02-05T05:18:03.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2024-02-05T05:18:03.883 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 377957122050 2024-02-05T05:18:03.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:18:04.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122051 -lt 377957122050 2024-02-05T05:18:04.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:04.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:04.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:04.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:04.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:18:04.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:04.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:04.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:18:05.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:05.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:05.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:18:05.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:18:05.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-02-05T05:18:05.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229736"' 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229736' 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:18:05.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229738"' 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229738' 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 229736 229738' 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229739: /' 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229736 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:18:05.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229741: /' 2024-02-05T05:18:08.334 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.334 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:08.334 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:08.335 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:18:08.336 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:08.337 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:08.337 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:08.337 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:08.338 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: start osd.9 2024-02-05T05:18:08.339 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:18:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:18:08.384 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:08.384 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:08.384 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:08.384 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:08.384 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:08.385 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: start osd.5 2024-02-05T05:18:08.386 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 2024-02-05T05:18:08.364+0000 7fb7e76144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 2024-02-05T05:18:08.367+0000 7fb7e76144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 2024-02-05T05:18:08.368+0000 7fb7e76144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:18:12.851 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 0 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 2024-02-05T05:18:08.882+0000 7fb7e76144c0 -1 Falling back to public interface 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 1 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.852 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 2 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 2024-02-05T05:18:11.665+0000 7fb7e76144c0 -1 osd.9 90 log_to_monitors true 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 3 2024-02-05T05:18:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.854 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:12.862 INFO:tasks.workunit.client.0.smithi188.stderr:229741: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 91 last_c229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:18:12.862 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 2024-02-05T05:18:08.410+0000 7ff991b944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 2024-02-05T05:18:08.412+0000 7ff991b944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 2024-02-05T05:18:08.413+0000 7ff991b944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.863 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 0 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 2024-02-05T05:18:08.928+0000 7ff991b944c0 -1 Falling back to public interface 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 1 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:12.864 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 2 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 2024-02-05T05:18:11.764+0000 7ff991b944c0 -1 osd.5 90 log_to_monitors true 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 3 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:13.749 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperlean_interval [88,90) [v2:127.0.0.1:6842/1226986301,v1:127.0.0.1:6843/1226986301] [v2:127.0.0.1:6844/1226986301,v1:127.0.0.1:6845/1226986301] exists,up 2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:13.750 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 1 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 2 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 3 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 4 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 5 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 6 2024-02-05T05:18:13.751 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 7 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 8 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: 9' 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673042 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673042 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042' 2024-02-05T05:18:13.752 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qas.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 4 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:229739: osd.5 up in weight 1 up_from 97 up_thru 83 down_at 91 last_clean_interval [83,90) [v2:127.0.0.1:6874/715672922,v1:127.0.0.1:6875/715672922] [v2:127.0.0.1:6876/715672922,v1:127.0.0.1:6877/715672922] exists,up a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 1 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 2 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 3 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 4 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 5 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 6 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 7 2024-02-05T05:18:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 8 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: 9' 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836573 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836573 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573' 2024-02-05T05:18:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //ho/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519' 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678035 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678035 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519 3-304942678035' 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:18:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519 3-304942678035 4-107374182463' 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827714 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827714 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519 3-304942678035 4-107374182463 5-416611827714' 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855412 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855412 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519 3-304942678035 4-107374182463 5-416611827714 6-150323855412' 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:18:15.466 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-02-05T05:18:15.466 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-02-05T05:18:15.466 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519 3-304942678035 4-107374182463 5-416611827714 6-150323855412 7-171798691883' 2024-02-05T05:18:15.466 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:15.466 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:18:16.291 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/cloneme/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:16.291 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044' 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520' 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678037 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678037 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520 3-304942678037' 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.292 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520 3-304942678037 4-107374182464' 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=416611827715 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 416611827715 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520 3-304942678037 4-107374182464 5-416611827715' 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855413 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855413 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520 3-304942678037 4-107374182464 5-416611827715 6-150323855413' 2024-02-05T05:18:16.293 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.294 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:18:16.306 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528361 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528361 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519 3-304942678035 4-107374182463 5-416611827714 6-150323855412 7-171798691883 8-193273528361' 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673042 2-64424509519 3-304942678035 4-107374182463 5-416611827714 6-150323855412 7-171798691883 8-193273528361 9-412316860418' 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-02-05T05:18:16.307 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.0 seq 21474836572 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673042 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673042 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673042 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673042' 2024-02-05T05:18:16.308 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.1 seq 42949673042 2024-02-05T05:18:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673042 -lt 42949673042 2024-02-05T05:18:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-02-05T05:18:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.cstats: seq=171798691884 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691884 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520 3-304942678037 4-107374182464 5-416611827715 6-150323855413 7-171798691884' 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520 3-304942678037 4-107374182464 5-416611827715 6-150323855413 7-171798691884 8-193273528362' 2024-02-05T05:18:16.849 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860419 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860419 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509520 3-304942678037 4-107374182464 5-416611827715 6-150323855413 7-171798691884 8-193273528362 9-412316860419' 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836573 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836573 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836573 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.0 seq 21474836573 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:16.850 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836573 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-02-05T05:18:16.851 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.1 seq 42949673044 2024-02-05T05:18:17.199 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_statlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-02-05T05:18:17.199 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.2 seq 64424509519 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509519 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678035 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678035 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678035 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678035' 2024-02-05T05:18:17.200 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.3 seq 304942678035 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678035 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.4 seq 107374182463 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182465 -lt 107374182463 2024-02-05T05:18:17.201 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:17.202 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827714 2024-02-05T05:18:17.202 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:17.202 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:18:17.202 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827714 2024-02-05T05:18:17.202 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:17.202 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827714 2024-02-05T05:18:18.051 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitis: ceph osd last-stat-seq 1 2024-02-05T05:18:18.051 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673044 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.2 seq 64424509520 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509520 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678037 2024-02-05T05:18:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678037 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678037 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678037' 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.3 seq 304942678037 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678037 -lt 304942678037 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-02-05T05:18:18.053 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-02-05T05:18:18.054 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.4 seq 107374182464 2024-02-05T05:18:18.054 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:18:18.054 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182465 -lt 107374182464 2024-02-05T05:18:18.054 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.054 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-416611827715 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtng osd.5 seq 416611827714' 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.5 seq 416611827714 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827714 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855412 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855412 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855412 2024-02-05T05:18:18.474 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.6 seq 150323855412 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855412 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.7 seq 171798691883 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691883 2024-02-05T05:18:18.475 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528361 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528361 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528361 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528361' 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.8 seq 193273528361 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:18:18.476 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528362 -lt 193273528361 2024-02-05T05:18:18.981 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.981 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:18:18.981 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-416611827715 2024-02-05T05:18:18.981 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.981 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=416611827715 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 416611827715' 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.5 seq 416611827715 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 416611827715 -lt 416611827715 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855413 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855413 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855413 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855413' 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.6 seq 150323855413 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:18:18.982 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855413 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691884 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691884 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691884 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691884' 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.7 seq 171798691884 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691885 -lt 171798691884 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-02-05T05:18:18.983 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:18.984 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:18:18.984 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut _stats: for s in $seqs 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860418 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860418 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860418' 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: waiting osd.9 seq 412316860418 2024-02-05T05:18:19.782 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860418 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:18:19.783 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:229741: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:18:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:229741: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:18:20.573 INFO:tasks.workunit.client.0.smithi188.stderr:-d - -f 2 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.8 seq 193273528362 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528363 -lt 193273528362 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-412316860419 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-412316860419 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860419 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 412316860419' 2024-02-05T05:18:20.574 INFO:tasks.workunit.client.0.smithi188.stderr:229739: waiting osd.9 seq 412316860419 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860419 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:20.575 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:229739: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:229739: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229738 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:18:20.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:18:20.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:18:20.577 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:18:20.577 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:18:20.577 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:18:20.577 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:18:20.577 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:18:20.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:18:20.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:18:20.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:18:20.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:18:20.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:20.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:20.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:20.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:21.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:21.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:18:21.388 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:21.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:45.873472+0000 '>' 2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:21.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:18:22.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:18:22.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:18:22.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:18:22.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:22.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:22.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:22.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:22.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:45.873472+0000 '>' 2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:22.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:18:23.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:18:23.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:18:23.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:18:23.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:23.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:23.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:23.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:24.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:45.873472+0000 '>' 2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:24.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:18:25.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:18:25.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:18:25.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:18:25.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:25.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:25.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:25.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:25.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:17:45.873472+0000 '>' 2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:25.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:18:26.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:18:26.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:18:26.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:18:26.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:26.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:26.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:26.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:26.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:18:21.832167+0000 '>' 2024-02-05T05:17:45.873472+0000 2024-02-05T05:18:26.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 233167"' 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 233167' 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 233168"' 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 233168' 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 233167 233168' 2024-02-05T05:18:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:18:26.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:18:26.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:18:26.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 233167 2024-02-05T05:18:26.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/233170: /' 2024-02-05T05:18:26.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:18:26.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/233172: /' 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:29.591 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: _ 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: hinfo_key 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: snapset 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:29.594 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:29.595 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:29.596 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:29.596 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:18:29.596 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:18:29.596 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:18:29.596 INFO:tasks.workunit.client.0.smithi188.stderr:233172: start osd.9 2024-02-05T05:18:29.687 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: _ 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: hinfo_key 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: snapset 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:18:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:18:29.690 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:29.691 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:18:29.692 INFO:tasks.workunit.client.0.smithi188.stderr:233170: start osd.5 2024-02-05T05:18:34.427 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:18:34.427 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:18:34.427 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 2024-02-05T05:18:29.715+0000 7f80b10104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 2024-02-05T05:18:29.718+0000 7f80b10104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 2024-02-05T05:18:29.719+0000 7f80b10104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.428 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 0 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 2024-02-05T05:18:30.233+0000 7f80b10104c0 -1 Falling back to public interface 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 1 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:34.429 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 2 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 2024-02-05T05:18:32.980+0000 7f80b10104c0 -1 osd.5 100 log_to_monitors true 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 3 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233170: osd.5 up in weight 1 up_from 105 up_thru 105 dowovery_ops 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 2024-02-05T05:18:29.621+0000 7fbd5bac94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 2024-02-05T05:18:29.624+0000 7fbd5bac94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 2024-02-05T05:18:29.625+0000 7fbd5bac94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:34.433 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 0 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 2024-02-05T05:18:30.147+0000 7fbd5bac94c0 -1 Falling back to public interface 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 1 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 2 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 2024-02-05T05:18:32.713+0000 7fbd5bac94c0 -1 osd.9 100 log_to_monitors true 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 3 2024-02-05T05:18:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:18:35.626 INFO:tasks.workunit.client.0.smithi188.stderr:233172: osd.9 up in weight 1 up_from 105 up_thru 0 down_at 101 last_clean_interval [96,100) [v2:127.0.0.1:6842/2525715841,v1:127.0.0.1:6843/2525715841] [v2:127.0.0.1:6844/2525715841,v1:127.0.0.1:6845/2525715841] exists,up 2ff15301-29c5-4bee-8cc3-e4164070278a 2024-02-05T05:18:35.626 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:35.626 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 1 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 2 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 3 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 4 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 5 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 6 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 7 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 8 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: 9' 2024-02-05T05:18:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049' 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clon_at 101 last_clean_interval [97,100) [v2:127.0.0.1:6874/1556114150,v1:127.0.0.1:6875/1556114150] [v2:127.0.0.1:6876/1556114150,v1:127.0.0.1:6877/1556114150] exists,up a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:35.629 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 1 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 2 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 3 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 4 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 5 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 6 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 7 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 8 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: 9' 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:35.631 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050' 2024-02-05T05:18:35.632 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.811 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/cne.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525' 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678042 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678042 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525 3-304942678042' 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182470 2024-02-05T05:18:36.812 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182470 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525 3-304942678042 4-107374182470' 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525 3-304942678042 4-107374182470 5-450971566082' 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855419 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855419 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525 3-304942678042 4-107374182470 5-450971566082 6-150323855419' 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.813 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691889 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691889 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525 3-304942678042 4-107374182470 5-450971566082 6-150323855419 7-171798691889' 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/lone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526' 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678043 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678043 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526 3-304942678043' 2024-02-05T05:18:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526 3-304942678043 4-107374182471' 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526 3-304942678043 4-107374182471 5-450971566083' 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-02-05T05:18:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526 3-304942678043 4-107374182471 5-450971566083 6-150323855418' 2024-02-05T05:18:36.816 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.816 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:18:36.816 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691890 2024-02-05T05:18:36.816 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691890 2024-02-05T05:18:36.816 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526 3-304942678043 4-107374182471 5-450971566083 6-150323855418 7-171798691890' 2024-02-05T05:18:36.816 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:36.816 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528368 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528368 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525 3-304942678042 4-107374182470 5-450971566082 6-150323855419 7-171798691889 8-193273528368' 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673049 2-64424509525 3-304942678042 4-107374182470 5-450971566082 6-150323855419 7-171798691889 8-193273528368 9-450971566083' 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:37.493 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.0 seq 21474836579 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836579 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-02-05T05:18:37.494 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.1 seq 42949673049 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673049 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:37.495 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/ceu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526 3-304942678043 4-107374182471 5-450971566083 6-150323855418 7-171798691890 8-193273528369' 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673050 2-64424509526 3-304942678043 4-107374182471 5-450971566083 6-150323855418 7-171798691890 8-193273528369 9-450971566082' 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:37.498 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.0 seq 21474836578 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836578 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-02-05T05:18:37.499 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-02-05T05:18:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.1 seq 42949673050 2024-02-05T05:18:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-02-05T05:18:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-02-05T05:18:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:38.377 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/phtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-02-05T05:18:38.377 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.377 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-02-05T05:18:38.377 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.2 seq 64424509525 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509525 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678042 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678042 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678042 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678042' 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.3 seq 304942678042 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:18:38.378 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678042 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182470 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182470 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182470 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182470' 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.4 seq 107374182470 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:18:38.379 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182471 -lt 107374182470 2024-02-05T05:18:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-450971566082 2024-02-05T05:18:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:18:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-450971566082 2024-02-05T05:18:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-02-05T05:18:38.456 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_statscephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-02-05T05:18:38.456 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.456 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-02-05T05:18:38.456 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-02-05T05:18:38.456 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.2 seq 64424509526 2024-02-05T05:18:38.456 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509526 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-304942678043 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-304942678043 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678043 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 304942678043' 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.3 seq 304942678043 2024-02-05T05:18:38.457 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678043 -lt 304942678043 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.4 seq 107374182471 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182471 -lt 107374182471 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-450971566083 2024-02-05T05:18:38.458 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:38.459 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:18:38.459 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-450971566083 2024-02-05T05:18:38.459 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:38.459 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_sta: echo 'waiting osd.5 seq 450971566082' 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.5 seq 450971566082 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566082 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855419 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:18:39.640 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855419 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855419 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855419' 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.6 seq 150323855419 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855419 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691889 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691889 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:39.641 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691889 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.7 seq 171798691889 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691891 -lt 171798691889 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528368 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528368 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528368 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.8 seq 193273528368 2024-02-05T05:18:39.642 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:18:39.643 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528369 -lt 193273528368 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:ts: echo 'waiting osd.5 seq 450971566083' 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.5 seq 450971566083 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566083 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:39.669 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.6 seq 150323855418 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855418 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691890 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691890 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691890 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.7 seq 171798691890 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:18:39.670 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691891 -lt 171798691890 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.8 seq 193273528369 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:18:39.671 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528369 -lt 193273528369 2024-02-05T05:18:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566083 2024-02-05T05:18:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:18:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566083 2024-02-05T05:18:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-02-05T05:18:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-02-05T05:18:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:233172: waiting osd.9 seq 450971566083 2024-02-05T05:18:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:18:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566083 2024-02-05T05:18:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:18:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:233172: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:18:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:233172: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:40.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:h:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566082 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566082 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566082' 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: waiting osd.9 seq 450971566082 2024-02-05T05:18:41.075 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566083 -lt 450971566082 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:18:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:233170: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:233170: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 233168 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:18:41.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:18:41.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:18:41.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-02-05T05:18:41.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:18:41.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:18:41.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:18:41.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236400"' 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236400' 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236401"' 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236401' 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 236400 236401' 2024-02-05T05:18:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:18:41.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:18:41.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:18:41.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236400 2024-02-05T05:18:41.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236403: /' 2024-02-05T05:18:41.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:41.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236405: /' 2024-02-05T05:18:43.971 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:43.971 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:43.971 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:18:43.971 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:43.974 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:43.974 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:43.974 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:43.974 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:18:43.974 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: start osd.5 2024-02-05T05:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:18:44.185 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:18:44.185 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:18:44.185 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:18:44.185 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:18:44.185 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:18:44.185 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:18:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:18:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:18:44.188 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:18:44.189 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: start osd.3 2024-02-05T05:18:44.190 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:18:48.754 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:18:48.754 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:18:48.754 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:48.754 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:48.754 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:48.754 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 2024-02-05T05:18:44.002+0000 7f54eb60c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 2024-02-05T05:18:44.004+0000 7f54eb60c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 2024-02-05T05:18:44.005+0000 7f54eb60c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 0 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 2024-02-05T05:18:44.530+0000 7f54eb60c4c0 -1 Falling back to public interface 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:48.755 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 1 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 2 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:48.756 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 2024-02-05T05:18:47.043+0000 7f54eb60c4c0 -1 osd.5 108 log_to_monitors true 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 3 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236405: osd.5 up in weight 1 up_from 113 up_thru 113 down_at 109 l236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 2024-02-05T05:18:44.212+0000 7fbc8448c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 2024-02-05T05:18:44.215+0000 7fbc8448c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:48.757 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 2024-02-05T05:18:44.216+0000 7fbc8448c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 0 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 2024-02-05T05:18:44.735+0000 7fbc8448c4c0 -1 Falling back to public interface 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:48.758 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 1 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 2 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 2024-02-05T05:18:47.631+0000 7fbc8448c4c0 -1 osd.3 108 log_to_monitors true 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:48.759 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:48.760 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:18:48.760 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 3 2024-02-05T05:18:48.760 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:48.760 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:18:49.509 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeast_clean_interval [105,108) [v2:127.0.0.1:6826/3856336303,v1:127.0.0.1:6827/3856336303] [v2:127.0.0.1:6828/3856336303,v1:127.0.0.1:6829/3856336303] exists,up a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:18:49.509 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:49.509 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:49.510 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 1 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 2 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 3 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 4 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 5 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 6 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 7 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 8 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: 9' 2024-02-05T05:18:49.511 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:49.512 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:49.512 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:49.512 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836583 2024-02-05T05:18:49.512 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836583 2024-02-05T05:18:49.512 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583' 2024-02-05T05:18:49.512 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:49.512 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:49.513 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-02-05T05:18:49.513 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-02-05T05:18:49.513 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053' 2024-02-05T05:18:49.513 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.646 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:50.646 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-02-05T05:18:50.646 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530' 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530 3-498216206338' 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-02-05T05:18:50.647 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530 3-498216206338 4-107374182474' 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530 3-498216206338 4-107374182474 5-485331304450' 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855423 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855423 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530 3-498216206338 4-107374182474 5-485331304450 6-150323855423' 2024-02-05T05:18:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.649 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:18:50.649 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-02-05T05:18:50.649 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-02-05T05:18:50.649 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530 3-498216206338 4-107374182474 5-485331304450 6-150323855423 7-171798691894' 2024-02-05T05:18:50.649 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.649 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:18:50.717 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtesrs.sh:985: wait_for_osd: sleep 1 2024-02-05T05:18:50.717 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:18:50.717 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:18:50.717 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 4 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: osd.3 up in weight 1 up_from 116 up_thru 116 down_at 109 last_clean_interval [71,108) [v2:127.0.0.1:6874/3794384375,v1:127.0.0.1:6875/3794384375] [v2:127.0.0.1:6876/3794384375,v1:127.0.0.1:6877/3794384375] exists,up 3e670ed6-5780-4b95-8c34-207ce6eba772 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:18:50.718 INFO:tasks.workunit.client.0.smithi188.stderr:236403: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 1 2024-02-05T05:18:50.719 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 2 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 3 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 4 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 5 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 6 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 7 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 8 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: 9' 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:18:50.720 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-02-05T05:18:50.721 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-02-05T05:18:50.721 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-02-05T05:18:50.721 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.348 INFO:tasks.workunit.client.0.smithi188.stderr:236t/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528372 2024-02-05T05:18:51.348 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528372 2024-02-05T05:18:51.348 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530 3-498216206338 4-107374182474 5-485331304450 6-150323855423 7-171798691894 8-193273528372' 2024-02-05T05:18:51.348 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566087 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566087 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673053 2-64424509530 3-498216206338 4-107374182474 5-485331304450 6-150323855423 7-171798691894 8-193273528372 9-450971566087' 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836583 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836583 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:51.349 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836583 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.0 seq 21474836583 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836583 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-02-05T05:18:51.350 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.1 seq 42949673053 2024-02-05T05:18:51.351 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:51.351 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673053 2024-02-05T05:18:51.351 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:51.351 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-02-05T05:18:51.351 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:51.351 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:51.800 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:18:51.800 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-02-05T05:18:51.800 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-02-05T05:18:51.800 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055' 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531' 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:18:51.801 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206339 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206339 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531 3-498216206339' 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531 3-498216206339 4-107374182476' 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.802 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531 3-498216206339 4-107374182476 5-485331304451' 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855424 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855424 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531 3-498216206339 4-107374182476 5-485331304451 6-150323855424' 2024-02-05T05:18:51.803 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:51.804 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:18:52.276 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.2 seq 64424509530 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509530 -lt 64424509530 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206338 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206338 2024-02-05T05:18:52.277 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206338' 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.3 seq 498216206338 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-02-05T05:18:52.278 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.4 seq 107374182474 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182474 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304450 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304450 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:52.279 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 485331304450' 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.5 seq 485331304450 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304450 -lt 485331304450 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855423 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:18:53.388 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855423 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855423 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.6 seq 150323855423 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855423 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-02-05T05:18:53.389 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.7 seq 171798691894 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691896 -lt 171798691894 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528372 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528372 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528372 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.8 seq 193273528372 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:18:53.390 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528374 -lt 193273528372 2024-02-05T05:18:53.611 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fflush_pg_stats: seq=171798691895 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691895 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531 3-498216206339 4-107374182476 5-485331304451 6-150323855424 7-171798691895' 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531 3-498216206339 4-107374182476 5-485331304451 6-150323855424 7-171798691895 8-193273528374' 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673055 2-64424509531 3-498216206339 4-107374182476 5-485331304451 6-150323855424 7-171798691895 8-193273528374 9-450971566088' 2024-02-05T05:18:53.612 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.0 seq 21474836584 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836584 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:18:53.613 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836584 2024-02-05T05:18:53.614 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:53.614 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-02-05T05:18:53.614 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:53.614 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:18:53.614 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-02-05T05:18:54.497 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:54.497 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-02-05T05:18:54.497 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-02-05T05:18:54.497 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.1 seq 42949673055 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673055 -lt 42949673055 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.2 seq 64424509531 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509531 2024-02-05T05:18:54.498 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-498216206339 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-498216206339 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206339 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 498216206339' 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.3 seq 498216206339 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206339 -lt 498216206339 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:18:54.499 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-02-05T05:18:54.500 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:54.500 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-02-05T05:18:54.500 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-02-05T05:18:54.500 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.4 seq 107374182476 2024-02-05T05:18:54.778 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtestlush_pg_stats: for s in $seqs 2024-02-05T05:18:54.778 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566087 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566087 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566087 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566087' 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: waiting osd.9 seq 450971566087 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566087 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:18:54.779 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:54.780 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:54.781 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:55.436 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:55.436 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:55.436 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:55.436 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:55.436 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:236405: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:236405: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:55.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:18:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:18:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182476 2024-02-05T05:18:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304451 2024-02-05T05:18:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304451 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 485331304451' 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.5 seq 485331304451 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304452 -lt 485331304451 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855424 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855424 2024-02-05T05:18:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855424 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855424' 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.6 seq 150323855424 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855425 -lt 150323855424 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691895 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691895 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691895 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-02-05T05:18:55.656 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.7 seq 171798691895 2024-02-05T05:18:55.657 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:18:55.657 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691896 -lt 171798691895 2024-02-05T05:18:55.657 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:56.797 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.8 seq 193273528374 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528374 -lt 193273528374 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566088 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:18:56.798 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566088 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566088' 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: waiting osd.9 seq 450971566088 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566088 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:18:56.799 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:18:56.800 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:18:56.800 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:18:56.800 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:18:56.800 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:18:57.129 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:18:57.129 INFO:tasks.workunit.client.0.smithi188.stderr:236403: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:236403: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236401 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:18:57.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:18:57.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:57.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:57.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:18:21.832167+0000 2024-02-05T05:18:57.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:18:57.942 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:18:57.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:18:21.832167+0000 2024-02-05T05:18:57.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:18:57.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:18:21.832167+0000 2024-02-05T05:18:57.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:18:57.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:18:57.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:18:57.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:18:57.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:57.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:57.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:57.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:58.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:18:21.832167+0000 '>' 2024-02-05T05:18:21.832167+0000 2024-02-05T05:18:58.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:18:59.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:18:59.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:18:59.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:18:59.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:18:59.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:18:59.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:18:59.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:18:59.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:18:21.832167+0000 '>' 2024-02-05T05:18:21.832167+0000 2024-02-05T05:18:59.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:19:00.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:19:00.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:19:00.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:19:00.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:19:00.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:19:00.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:19:00.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:18:58.608736+0000 '>' 2024-02-05T05:18:21.832167+0000 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239852"' 2024-02-05T05:19:00.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239852' 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239854"' 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239854' 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 239852 239854' 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239852 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239855: /' 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:19:00.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239857: /' 2024-02-05T05:19:03.645 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:19:03.645 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:19:03.645 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:19:03.645 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:19:03.645 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:19:03.646 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: _ 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: hinfo_key 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: snapset 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:19:03.647 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:19:03.648 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:19:03.648 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:19:03.648 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:19:03.648 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:19:03.648 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:03.649 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:19:03.650 INFO:tasks.workunit.client.0.smithi188.stderr:239855: start osd.3 2024-02-05T05:19:04.196 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:19:04.196 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:19:04.196 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:19:04.196 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:19:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: _ 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: hinfo_key 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: snapset 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:19:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:19:04.199 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:19:04.200 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:19:04.201 INFO:tasks.workunit.client.0.smithi188.stderr:239857: start osd.5 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 2024-02-05T05:19:03.676+0000 7f962868a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 2024-02-05T05:19:03.678+0000 7f962868a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 2024-02-05T05:19:03.680+0000 7f962868a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:19:07.744 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 0 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 2024-02-05T05:19:04.202+0000 7f962868a4c0 -1 Falling back to public interface 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 1 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 2 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 2024-02-05T05:19:06.560+0000 7f962868a4c0 -1 osd.3 117 log_to_monitors true 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:07.747 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:07.747 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:19:07.747 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 3 2024-02-05T05:19:07.747 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 2024-02-05T05:19:04.223+0000 7fe76cbbc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 2024-02-05T05:19:04.225+0000 7fe76cbbc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 2024-02-05T05:19:04.226+0000 7fe76cbbc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:19:08.762 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 0 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 2024-02-05T05:19:04.747+0000 7fe76cbbc4c0 -1 Falling back to public interface 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:08.763 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 1 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 2 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 2024-02-05T05:19:07.582+0000 7fe76cbbc4c0 -1 osd.5 117 log_to_monitors true 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:08.764 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:08.765 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:19:08.765 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 3 2024-02-05T05:19:08.765 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:08.765 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:19:09.021 INFO:tasks.workunit.client.0.smithi188.stderr:239857: osd.5 up in weight 1 up_from 123 up_thru 113 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:19:09.021 INFO:tasks.workunit.client.0.smithi188.stderr:239855: osd.3 up in weight 1 up_from 121 up_thru 116 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6826/3018513086,v1:127.0.0.1:6827/3018513086] [v2:127.0.0.1:6828/3018513086,v1:127.0.0.1:6829/3018513086] exists,up 3e670ed6-5780-4b95-8c34-207ce6eba772 2024-02-05T05:19:09.021 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:19:09.021 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:19:09.022 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 1 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 2 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 3 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 4 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 5 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 6 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 7 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 8 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: 9' 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:19:09.023 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836589 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836589 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589' 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-02-05T05:19:09.024 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059' 2024-02-05T05:19:09.465 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0n_at 118 last_clean_interval [113,117) [v2:127.0.0.1:6874/2889283103,v1:127.0.0.1:6875/2889283103] [v2:127.0.0.1:6876/2889283103,v1:127.0.0.1:6877/2889283103] exists,up a0f6513a-323f-47de-9504-1ea4e7d0594f 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:19:09.466 INFO:tasks.workunit.client.0.smithi188.stderr:239857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 1 2024-02-05T05:19:09.467 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 2 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 3 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 4 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 5 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 6 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 7 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 8 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: 9' 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.468 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836590 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836590 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590' 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060' 2024-02-05T05:19:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.927 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest//qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536' 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536 3-519691042818' 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:19:09.928 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536 3-519691042818 4-107374182480' 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536 3-519691042818 4-107374182480 5-528280977410' 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855429 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855429 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536 3-519691042818 4-107374182480 5-528280977410 6-150323855429' 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:19:09.929 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691900 2024-02-05T05:19:09.930 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-02-05T05:19:09.930 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536 3-519691042818 4-107374182480 5-528280977410 6-150323855429 7-171798691900' 2024-02-05T05:19:09.930 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:10.724 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:19:10.724 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-02-05T05:19:10.724 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-02-05T05:19:10.724 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537' 2024-02-05T05:19:10.724 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:10.724 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042819 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042819 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537 3-519691042819' 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537 3-519691042819 4-107374182481' 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977411 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977411 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537 3-519691042819 4-107374182481 5-528280977411' 2024-02-05T05:19:10.725 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855430 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855430 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537 3-519691042819 4-107374182481 5-528280977411 6-150323855430' 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691901 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691901 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537 3-519691042819 4-107374182481 5-528280977411 6-150323855430 7-171798691901' 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:10.726 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528379 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528379 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537 3-519691042819 4-107374182481 5-528280977411 6-150323855430 7-171798691901 8-193273528379' 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836590 1-42949673060 2-64424509537 3-519691042819 4-107374182481 5-528280977411 6-150323855430 7-171798691901 8-193273528379 9-450971566094' 2024-02-05T05:19:11.394 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836590 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836590 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836590 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.0 seq 21474836590 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836590 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-02-05T05:19:11.395 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.1 seq 42949673060 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673060 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:11.396 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntua/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528378 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528378 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536 3-519691042818 4-107374182480 5-528280977410 6-150323855429 7-171798691900 8-193273528378' 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566093 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566093 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673059 2-64424509536 3-519691042818 4-107374182480 5-528280977410 6-150323855429 7-171798691900 8-193273528378 9-450971566093' 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:12.337 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836589 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836589 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836589 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.0 seq 21474836589 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836588 -lt 21474836589 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:19:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836589 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.1 seq 42949673059 2024-02-05T05:19:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:19:12.346 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/c/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-02-05T05:19:12.346 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:12.346 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-02-05T05:19:12.346 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.2 seq 64424509537 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-519691042819 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-519691042819 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042819 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 519691042819' 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.3 seq 519691042819 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042819 -lt 519691042819 2024-02-05T05:19:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.4 seq 107374182481 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182481 -lt 107374182481 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977411 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:19:12.349 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977411 2024-02-05T05:19:12.349 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:12.349 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977411 2024-02-05T05:19:13.231 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673059 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.2 seq 64424509536 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509536 2024-02-05T05:19:13.232 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-519691042818 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-519691042818 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 519691042818' 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.3 seq 519691042818 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042820 -lt 519691042818 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:13.233 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.4 seq 107374182480 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182480 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:13.234 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-528280977410 2024-02-05T05:19:13.603 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22ats: echo 'waiting osd.5 seq 528280977411' 2024-02-05T05:19:13.603 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.5 seq 528280977411 2024-02-05T05:19:13.603 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:19:13.603 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977411 -lt 528280977411 2024-02-05T05:19:13.603 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:13.603 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855430 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855430 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855430 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855430' 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.6 seq 150323855430 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855430 -lt 150323855430 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691901 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:13.604 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691901 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691901 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691901' 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.7 seq 171798691901 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691902 -lt 171798691901 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528379 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528379 2024-02-05T05:19:13.605 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:13.606 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528379 2024-02-05T05:19:13.606 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-02-05T05:19:13.606 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.8 seq 193273528379 2024-02-05T05:19:13.606 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:19:13.606 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528380 -lt 193273528379 2024-02-05T05:19:14.188 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.11: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:14.189 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:19:14.189 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-528280977410 2024-02-05T05:19:14.189 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:14.189 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-02-05T05:19:14.189 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 528280977410' 2024-02-05T05:19:14.189 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.5 seq 528280977410 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977411 -lt 528280977410 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855429 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855429 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855429 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.6 seq 150323855429 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855430 -lt 150323855429 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:14.190 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.7 seq 171798691900 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691902 -lt 171798691900 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528378 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528378 2024-02-05T05:19:14.191 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.cliensh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566094 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566094 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-02-05T05:19:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566094' 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: waiting osd.9 seq 450971566094 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566094 -lt 450971566094 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:19:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:19:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:19:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:19:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:239857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:19:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:19:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:19:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:239857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:19:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:t.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528378 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528378' 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.8 seq 193273528378 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528380 -lt 193273528378 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566093 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:19:15.873 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566093 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566093 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566093' 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: waiting osd.9 seq 450971566093 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566095 -lt 450971566093 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:19:15.874 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:239855: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:19:15.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239854 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:19:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:19:15.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:19:15.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:19:15.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:19:16.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:19:16.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:19:16.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:19:16.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:19:16.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:19:16.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:19:16.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:19:16.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:19:16.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:19:16.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:19:16.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:19:16.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:19:16.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:19:16.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:19:16.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:19:16.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:16.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:16.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:19:16.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:19:16.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:19:16.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:19:16.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:19:16.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:19:16.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:19:16.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:19:16.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:19:16.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:19:16.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:19:16.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:19:16.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:19:16.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:19:16.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:19:16.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:19:16.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:19:16.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:19:16.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:19:16.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:19:16.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:19:16.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:16.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:16.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:19:16.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:19:16.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:19:16.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:19:16.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:19:16.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:16.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:16.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-02-05T05:19:16.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-02-05T05:19:16.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-02-05T05:19:16.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:19:16.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:19:16.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:19:16.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:19:16.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:19:16.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:16.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:19:16.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:19:16.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:19:16.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:19:16.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:19:16.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:19:16.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:19:16.712 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:19:16.713 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:16.791 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:16.792 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:19:16.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:19:16.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:19:16.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-02-05T05:19:16.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:19:16.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:19:16.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:19:16.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:19:16.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:19:16.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:19:17.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:19:17.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:19:17.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:19:17.680 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:19:17.680 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:17.680 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:17.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:17.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:19:17.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:19:17.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-02-05T05:19:17.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:19:17.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-02-05T05:19:17.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:19:17.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:19:17.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:19:17.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:19:17.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:19:17.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:19:17.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:19:17.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:19:17.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:19:17.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:19:17.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28b8ae63-8662-40b8-bad3-8a1ffdec5663 2024-02-05T05:19:17.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 28b8ae63-8662-40b8-bad3-8a1ffdec5663' 2024-02-05T05:19:17.717 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 28b8ae63-8662-40b8-bad3-8a1ffdec5663 2024-02-05T05:19:17.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:19:17.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVb8BlxxCRKxAAbwnkk0Xrzx7Tp+nBTRNgFQ== 2024-02-05T05:19:17.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVb8BlxxCRKxAAbwnkk0Xrzx7Tp+nBTRNgFQ=="}' 2024-02-05T05:19:17.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28b8ae63-8662-40b8-bad3-8a1ffdec5663 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:19:17.975 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:19:17.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:19:17.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVb8BlxxCRKxAAbwnkk0Xrzx7Tp+nBTRNgFQ== --osd-uuid 28b8ae63-8662-40b8-bad3-8a1ffdec5663 2024-02-05T05:19:18.015 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:18.013+0000 7f75ad01c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:18.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:18.026+0000 7f75ad01c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:18.028 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:18.027+0000 7f75ad01c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:18.029 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:18.027+0000 7f75ad01c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:19:18.029 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:18.027+0000 7f75ad01c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:19:18.029 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:18.027+0000 7f75ad01c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:19:18.128 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:18.126+0000 7f75ad01c4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:19:23.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:19:23.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:19:23.656 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:19:23.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:19:23.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:19:24.003 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:19:24.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:19:24.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:19:24.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:19:24.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:19:24.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:19:24.031 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:24.027+0000 7f280588a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:24.032 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:24.029+0000 7f280588a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:24.032 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:24.030+0000 7f280588a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:19:24.269 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:19:24.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:24.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:24.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:24.550 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:24.549+0000 7f280588a4c0 -1 Falling back to public interface 2024-02-05T05:19:25.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:25.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:25.536 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:19:25.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:19:25.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:25.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:25.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:26.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:26.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:26.815 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:19:26.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:19:26.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:26.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:26.935 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:26.934+0000 7f280588a4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:19:27.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:28.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:28.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:28.120 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:19:28.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:19:28.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:28.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:28.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:29.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:29.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:29.399 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:19:29.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:19:29.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:29.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:29.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:30.782 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:19:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:19:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:31.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:32.061 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:19:32.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:19:32.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:32.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:32.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:33.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:33.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:33.342 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:19:33.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:19:33.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:33.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:33.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:34.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:34.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:34.622 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:19:34.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:19:34.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:34.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:34.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:35.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:35.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:35.966 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:19:35.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:19:35.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:35.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:36.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:37.245 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:19:37.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:19:37.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:37.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:37.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:38.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:38.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:38.627 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:19:38.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:19:38.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:38.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:38.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:39.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:39.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:39.935 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:19:39.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:19:39.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:39.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:40.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:41.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:41.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:41.224 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:19:41.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:19:41.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:41.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:41.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:42.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:42.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:42.702 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:19:42.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:19:42.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:42.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:43.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:44.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:44.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:44.404 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:19:44.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:19:44.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:44.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:45.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:46.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:46.290 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:19:46.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:46.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:19:46.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:46.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:46.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:47.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:47.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:47.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:19:47.999 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:19:48.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:48.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:48.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:49.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:49.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:49.427 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:19:49.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:19:49.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:49.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:49.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:50.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:50.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:50.705 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:19:50.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:19:50.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:50.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:19:50.976 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3722433428,v1:127.0.0.1:6803/3722433428] [v2:127.0.0.1:6804/3722433428,v1:127.0.0.1:6805/3722433428] exists,up 28b8ae63-8662-40b8-bad3-8a1ffdec5663 2024-02-05T05:19:50.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:19:50.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:19:50.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:19:50.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:19:50.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-02-05T05:19:50.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:19:50.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:19:50.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d69f9723-9a1a-414d-9ed3-496bcbd94a91 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d69f9723-9a1a-414d-9ed3-496bcbd94a91' 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 d69f9723-9a1a-414d-9ed3-496bcbd94a91 2024-02-05T05:19:50.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:19:50.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2b8BlF1NeOxAAQFPEG4YWrpO6J5U5I6FI0Q== 2024-02-05T05:19:50.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2b8BlF1NeOxAAQFPEG4YWrpO6J5U5I6FI0Q=="}' 2024-02-05T05:19:50.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d69f9723-9a1a-414d-9ed3-496bcbd94a91 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:19:51.348 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:19:51.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:19:51.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2b8BlF1NeOxAAQFPEG4YWrpO6J5U5I6FI0Q== --osd-uuid d69f9723-9a1a-414d-9ed3-496bcbd94a91 2024-02-05T05:19:51.387 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:51.385+0000 7fc1a63f94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:51.432 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:51.430+0000 7fc1a63f94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:51.434 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:51.431+0000 7fc1a63f94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:51.434 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:51.432+0000 7fc1a63f94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:19:51.434 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:51.432+0000 7fc1a63f94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:19:51.434 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:51.432+0000 7fc1a63f94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:19:51.518 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:51.516+0000 7fc1a63f94c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:19:57.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:19:57.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:19:57.223 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:19:57.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:19:57.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:19:57.570 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:19:57.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:19:57.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:19:57.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:19:57.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:19:57.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:19:57.596 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:57.594+0000 7f68ff12b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:57.598 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:57.597+0000 7f68ff12b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:57.599 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:57.598+0000 7f68ff12b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:19:57.841 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:19:57.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:57.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:58.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:19:58.117 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:19:58.116+0000 7f68ff12b4c0 -1 Falling back to public interface 2024-02-05T05:19:59.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:19:59.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:19:59.110 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:19:59.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:19:59.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:19:59.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:19:59.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:00.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:00.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:00.381 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:20:00.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:20:00.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:00.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:00.448 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:00.447+0000 7f68ff12b4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:20:00.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:01.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:01.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:01.659 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:20:01.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:20:01.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:01.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:02.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:02.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:02.946 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:20:02.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:20:02.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:02.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:03.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:04.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:04.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:04.229 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:20:04.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:20:04.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:04.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:04.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:05.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:05.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:05.595 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:20:05.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:20:05.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:05.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:05.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:06.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:06.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:06.937 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:20:06.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:20:06.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:06.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:07.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:08.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:08.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:08.235 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:20:08.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:20:08.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:08.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:08.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:09.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:09.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:09.559 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:20:09.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:20:09.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:09.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:09.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:10.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:10.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:10.839 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:20:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:20:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:11.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:12.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:12.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:12.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:20:12.118 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:20:12.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:12.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:12.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:13.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:13.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:13.712 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:20:13.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:20:13.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:13.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:14.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:15.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:15.180 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:20:15.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:20:15.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:15.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:15.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:16.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:16.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:16.882 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:20:16.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:20:16.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:16.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:17.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:18.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:18.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:18.530 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:20:18.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:20:18.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:18.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:19.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:20.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:20.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:20.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:20:20.081 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:20:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:20.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:20.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:21.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:21.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:21.810 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:20:21.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:20:21.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:21.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:22.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:23.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:23.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:23.612 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:20:23.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:20:23.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:23.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:20:23.900 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1630950328,v1:127.0.0.1:6811/1630950328] [v2:127.0.0.1:6812/1630950328,v1:127.0.0.1:6813/1630950328] exists,up d69f9723-9a1a-414d-9ed3-496bcbd94a91 2024-02-05T05:20:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:20:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:20:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:20:23.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:20:23.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:20:23.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f84e20c0-c740-4bb6-a547-d6bf34abf8bf 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f84e20c0-c740-4bb6-a547-d6bf34abf8bf' 2024-02-05T05:20:23.907 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 f84e20c0-c740-4bb6-a547-d6bf34abf8bf 2024-02-05T05:20:23.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:20:23.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXcMBlwW/mNhAACY+XRtaY3tTmKXtcI3u3LQ== 2024-02-05T05:20:23.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXcMBlwW/mNhAACY+XRtaY3tTmKXtcI3u3LQ=="}' 2024-02-05T05:20:23.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f84e20c0-c740-4bb6-a547-d6bf34abf8bf -i td/osd-scrub-repair/2/new.json 2024-02-05T05:20:24.248 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:20:24.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:20:24.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXcMBlwW/mNhAACY+XRtaY3tTmKXtcI3u3LQ== --osd-uuid f84e20c0-c740-4bb6-a547-d6bf34abf8bf 2024-02-05T05:20:24.286 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:24.284+0000 7f0b5000c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:24.331 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:24.329+0000 7f0b5000c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:24.331 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:24.330+0000 7f0b5000c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:24.332 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:24.330+0000 7f0b5000c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:20:24.332 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:24.331+0000 7f0b5000c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:20:24.332 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:24.331+0000 7f0b5000c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:20:24.398 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:24.396+0000 7f0b5000c4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:20:30.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:20:30.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:20:30.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:20:30.082 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:20:30.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:20:30.425 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:20:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:20:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:20:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:20:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:20:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:20:30.451 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:30.449+0000 7f2b90ba94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:30.453 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:30.451+0000 7f2b90ba94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:30.454 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:30.453+0000 7f2b90ba94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:20:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:20:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:20:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:20:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:20:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:20:30.695 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:20:30.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:30.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:30.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:30.977 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:30.976+0000 7f2b90ba94c0 -1 Falling back to public interface 2024-02-05T05:20:31.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:31.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:31.960 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:20:31.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:20:31.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:31.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:32.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:33.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:33.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:33.236 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:20:33.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:20:33.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:33.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:33.355 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:33.354+0000 7f2b90ba94c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:20:33.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:34.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:34.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:34.557 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:20:34.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:20:34.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:34.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:34.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:35.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:35.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:35.838 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:20:35.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:20:35.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:35.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:36.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:37.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:37.168 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:20:37.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:37.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:20:37.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:37.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:37.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:38.445 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:20:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:20:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:38.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:39.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:39.812 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:20:39.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:39.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:20:39.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:39.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:40.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:41.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:41.128 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:20:41.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:41.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:20:41.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:41.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:41.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:42.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:42.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:42.405 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:20:42.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:20:42.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:42.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:42.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:43.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:43.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:43.682 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:20:43.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:20:43.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:43.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:44.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:45.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:45.072 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:20:45.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:45.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:20:45.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:45.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:45.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:46.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:46.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:46.667 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:20:46.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:20:46.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:46.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:47.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:48.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:48.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:48.349 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:20:48.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:20:48.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:48.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:48.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:49.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:49.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:49.686 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:20:49.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:20:49.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:49.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:50.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:51.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:51.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:51.172 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:20:51.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:20:51.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:51.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:51.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:52.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:52.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:52.794 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:20:52.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:20:52.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:52.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:53.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:54.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:54.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:54.076 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:20:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:20:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:54.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:54.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:55.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:55.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:55.590 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:20:55.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:20:55.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:55.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:55.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:20:56.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:20:56.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:20:56.883 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:20:56.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:20:56.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:20:56.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:20:57.154 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1475472842,v1:127.0.0.1:6819/1475472842] [v2:127.0.0.1:6820/1475472842,v1:127.0.0.1:6821/1475472842] exists,up f84e20c0-c740-4bb6-a547-d6bf34abf8bf 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:20:57.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:20:57.161 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:20:57.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:20:57.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f68a09e8-d5b4-4da0-8457-46288076ddeb 2024-02-05T05:20:57.163 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 f68a09e8-d5b4-4da0-8457-46288076ddeb 2024-02-05T05:20:57.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f68a09e8-d5b4-4da0-8457-46288076ddeb' 2024-02-05T05:20:57.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:20:57.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5cMBlalF1ChAAZ1lJwo3O3e4Jm83Nu1yfRQ== 2024-02-05T05:20:57.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5cMBlalF1ChAAZ1lJwo3O3e4Jm83Nu1yfRQ=="}' 2024-02-05T05:20:57.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f68a09e8-d5b4-4da0-8457-46288076ddeb -i td/osd-scrub-repair/3/new.json 2024-02-05T05:20:57.608 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:20:57.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T05:20:57.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5cMBlalF1ChAAZ1lJwo3O3e4Jm83Nu1yfRQ== --osd-uuid f68a09e8-d5b4-4da0-8457-46288076ddeb 2024-02-05T05:20:57.648 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:57.647+0000 7fd3ac3ab4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:57.710 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:57.709+0000 7fd3ac3ab4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:57.711 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:57.710+0000 7fd3ac3ab4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:20:57.711 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:57.710+0000 7fd3ac3ab4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:20:57.711 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:57.710+0000 7fd3ac3ab4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:20:57.711 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:57.710+0000 7fd3ac3ab4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T05:20:57.793 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:20:57.792+0000 7fd3ac3ab4c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T05:21:03.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T05:21:03.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:21:03.717 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T05:21:03.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T05:21:03.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:21:04.063 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T05:21:04.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T05:21:04.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:21:04.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:21:04.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:21:04.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:21:04.089 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:04.088+0000 7f8c09e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:04.091 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:04.090+0000 7f8c09e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:04.092 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:04.091+0000 7f8c09e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T05:21:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:21:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:21:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:21:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:21:04.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:04.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:21:04.334 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:21:04.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:04.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:04.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:04.614 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:04.613+0000 7f8c09e2a4c0 -1 Falling back to public interface 2024-02-05T05:21:05.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:05.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:05.600 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:21:05.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:21:05.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:05.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:05.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:06.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:06.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:06.885 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:21:06.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:21:06.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:06.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:06.962 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:06.960+0000 7f8c09e2a4c0 -1 osd.3 0 log_to_monitors true 2024-02-05T05:21:07.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:08.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:08.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:08.180 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:21:08.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:21:08.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:08.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:09.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:09.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:09.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:21:09.603 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:21:09.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:09.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:10.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:11.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:11.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:11.044 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:21:11.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:21:11.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:11.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:11.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:12.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:12.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:12.322 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:21:12.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:21:12.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:12.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:12.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:13.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:13.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:13.609 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:21:13.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:21:13.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:13.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:14.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:15.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:15.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:15.113 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:21:15.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:21:15.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:15.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:15.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:16.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:16.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:16.391 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:21:16.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:21:16.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:16.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:16.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:17.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:17.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:17.786 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:21:17.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:21:17.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:17.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:18.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:19.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:19.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:19.190 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:21:19.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:21:19.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:19.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:19.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:20.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:20.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:20.469 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:21:20.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:21:20.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:20.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:20.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:21.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:21.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:21.937 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:21:21.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:21:21.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:21.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:22.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:23.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:23.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:23.585 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:21:23.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:21:23.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:23.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:23.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:24.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:24.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:24.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:21:24.996 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:21:24.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:24.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:25.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:26.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:26.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:26.644 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:21:26.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:21:26.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:26.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:27.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:28.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:28.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:28.379 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:21:28.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:21:28.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:28.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:29.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:30.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:30.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:30.048 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:21:30.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:21:30.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:30.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:30.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:31.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:31.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:31.492 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:21:31.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:21:31.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:31.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:21:31.766 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3444823632,v1:127.0.0.1:6827/3444823632] [v2:127.0.0.1:6828/3444823632,v1:127.0.0.1:6829/3444823632] exists,up f68a09e8-d5b4-4da0-8457-46288076ddeb 2024-02-05T05:21:31.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:21:31.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:21:31.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:21:31.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:21:31.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-02-05T05:21:31.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:21:31.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:21:31.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-02-05T05:21:31.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:21:31.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-02-05T05:21:31.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:21:31.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:21:31.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:21:31.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:21:31.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-02-05T05:21:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-02-05T05:21:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:21:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:21:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:21:31.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:21:31.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:21:31.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc1ab506-0e0f-4117-a6d4-e7e844497d08 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 bc1ab506-0e0f-4117-a6d4-e7e844497d08' 2024-02-05T05:21:31.778 INFO:tasks.workunit.client.0.smithi188.stdout:add osd4 bc1ab506-0e0f-4117-a6d4-e7e844497d08 2024-02-05T05:21:31.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:21:31.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbcMBlLg/RLhAAiS1TZcKI4D2LiqtXvomWGg== 2024-02-05T05:21:31.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbcMBlLg/RLhAAiS1TZcKI4D2LiqtXvomWGg=="}' 2024-02-05T05:21:31.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc1ab506-0e0f-4117-a6d4-e7e844497d08 -i td/osd-scrub-repair/4/new.json 2024-02-05T05:21:32.104 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:21:32.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-02-05T05:21:32.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbcMBlLg/RLhAAiS1TZcKI4D2LiqtXvomWGg== --osd-uuid bc1ab506-0e0f-4117-a6d4-e7e844497d08 2024-02-05T05:21:32.144 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:32.142+0000 7f9ec874a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:32.187+0000 7f9ec874a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:32.189 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:32.188+0000 7f9ec874a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:32.189 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:32.188+0000 7f9ec874a4c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:21:32.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:32.188+0000 7f9ec874a4c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:21:32.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:32.188+0000 7f9ec874a4c0 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-02-05T05:21:32.272 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:32.270+0000 7f9ec874a4c0 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-02-05T05:21:38.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-02-05T05:21:38.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:21:38.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-02-05T05:21:38.309 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd4 key to auth repository 2024-02-05T05:21:38.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:21:38.658 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.4 2024-02-05T05:21:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-02-05T05:21:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:21:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:21:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:21:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:21:38.684 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:38.683+0000 7fc7f26cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:38.686 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:38.685+0000 7fc7f26cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:38.687 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:38.686+0000 7fc7f26cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:21:38.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-02-05T05:21:38.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:21:38.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-02-05T05:21:38.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:21:38.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:21:38.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:38.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:21:38.926 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:21:38.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:38.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:39.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:39.213 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:39.212+0000 7fc7f26cf4c0 -1 Falling back to public interface 2024-02-05T05:21:40.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:40.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:40.191 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:21:40.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:21:40.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:40.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:40.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:41.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:41.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:41.467 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:21:41.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:21:41.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:41.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:41.581 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:21:41.579+0000 7fc7f26cf4c0 -1 osd.4 0 log_to_monitors true 2024-02-05T05:21:41.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:42.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:42.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:42.791 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:21:42.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:21:42.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:42.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:43.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:21:44.071 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:21:44.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:44.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:45.352 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:21:45.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:21:45.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:45.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:45.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:46.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:46.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:46.631 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:21:46.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:21:46.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:46.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:46.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:47.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:47.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:47.915 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:21:47.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:21:47.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:47.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:48.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:49.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:49.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:49.255 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:21:49.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:21:49.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:49.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:50.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:50.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:50.657 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:21:50.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:21:50.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:50.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:50.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:51.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:51.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:51.936 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:21:51.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:21:51.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:51.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:52.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:53.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:53.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:53.265 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:21:53.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:21:53.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:53.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:53.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:54.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:54.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:54.798 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:21:54.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:21:54.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:54.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:55.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:56.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:56.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:56.097 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:21:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:21:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:56.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:57.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:57.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:57.993 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:21:57.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:21:57.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:57.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:21:58.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:21:59.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:21:59.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:21:59.788 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:21:59.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:21:59.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:21:59.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:22:00.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:01.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:01.641 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:22:01.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:01.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:22:01.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:01.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:22:02.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:03.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:03.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:03.072 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:22:03.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:22:03.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:03.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:22:03.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:04.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:04.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:04.881 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:22:04.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:22:04.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:04.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-02-05T05:22:05.252 INFO:tasks.workunit.client.0.smithi188.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/242970547,v1:127.0.0.1:6835/242970547] [v2:127.0.0.1:6836/242970547,v1:127.0.0.1:6837/242970547] exists,up bc1ab506-0e0f-4117-a6d4-e7e844497d08 2024-02-05T05:22:05.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:22:05.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:22:05.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:22:05.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:22:05.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c' 2024-02-05T05:22:05.260 INFO:tasks.workunit.client.0.smithi188.stdout:add osd5 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:22:05.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:22:05.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9cMBlKhY9EBAAyikxzqKpoFnaGMwTlZawQg== 2024-02-05T05:22:05.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9cMBlKhY9EBAAyikxzqKpoFnaGMwTlZawQg=="}' 2024-02-05T05:22:05.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c -i td/osd-scrub-repair/5/new.json 2024-02-05T05:22:05.605 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:22:05.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-02-05T05:22:05.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9cMBlKhY9EBAAyikxzqKpoFnaGMwTlZawQg== --osd-uuid 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:22:05.645 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:05.643+0000 7f2eda89a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:05.656 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:05.654+0000 7f2eda89a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:05.657 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:05.656+0000 7f2eda89a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:05.658 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:05.656+0000 7f2eda89a4c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:22:05.658 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:05.656+0000 7f2eda89a4c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:22:05.658 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:05.656+0000 7f2eda89a4c0 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-02-05T05:22:05.800 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:05.797+0000 7f2eda89a4c0 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-02-05T05:22:11.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-02-05T05:22:11.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:22:11.747 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd5 key to auth repository 2024-02-05T05:22:11.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-02-05T05:22:11.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:22:12.092 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.5 2024-02-05T05:22:12.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-02-05T05:22:12.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:22:12.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:22:12.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:22:12.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:22:12.118 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:12.116+0000 7f75fb80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:12.119 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:12.118+0000 7f75fb80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:12.120 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:12.119+0000 7f75fb80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:22:12.361 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:22:12.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:12.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:12.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:12.644 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:12.643+0000 7f75fb80e4c0 -1 Falling back to public interface 2024-02-05T05:22:13.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:13.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:13.632 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:22:13.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:22:13.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:13.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:13.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:14.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:14.911 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:22:14.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:14.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:22:14.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:14.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:15.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:15.026+0000 7f75fb80e4c0 -1 osd.5 0 log_to_monitors true 2024-02-05T05:22:15.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:16.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:16.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:16.238 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:22:16.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:22:16.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:16.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:16.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:17.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:17.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:17.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:18.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:19.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:19.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:19.073 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:22:19.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:22:19.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:19.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:19.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:20.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:20.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:20.353 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:22:20.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:22:20.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:20.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:20.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:21.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:21.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:21.633 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:22:21.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:22:21.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:21.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:22.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:23.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:23.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:23.090 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:22:23.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:22:23.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:23.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:23.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:24.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:24.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:24.370 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:22:24.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:22:24.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:24.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:24.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:25.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:25.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:25.649 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:22:25.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:22:25.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:25.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:26.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:27.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:27.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:27.111 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:22:27.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:22:27.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:27.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:27.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:28.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:28.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:28.388 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:22:28.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:22:28.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:28.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:29.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:30.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:30.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:30.243 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:22:30.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:22:30.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:30.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:30.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:31.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:31.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:31.955 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:22:31.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:22:31.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:31.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:32.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:33.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:33.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:33.374 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:22:33.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:22:33.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:33.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:33.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:34.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:34.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:34.843 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:22:34.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:22:34.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:34.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:35.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:36.690 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:22:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:22:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:37.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:38.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:38.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:38.143 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:22:38.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:22:38.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:38.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:22:38.439 INFO:tasks.workunit.client.0.smithi188.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3409905654,v1:127.0.0.1:6843/3409905654] [v2:127.0.0.1:6844/3409905654,v1:127.0.0.1:6845/3409905654] exists,up 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:22:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:22:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:22:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:22:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:22:38.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:22:38.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:22:38.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:22:38.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:22:38.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:22:38.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:22:38.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:22:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58da60aa-08bb-41f5-8504-2fa1f93b3cfd 2024-02-05T05:22:38.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 58da60aa-08bb-41f5-8504-2fa1f93b3cfd' 2024-02-05T05:22:38.449 INFO:tasks.workunit.client.0.smithi188.stdout:add osd6 58da60aa-08bb-41f5-8504-2fa1f93b3cfd 2024-02-05T05:22:38.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:22:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCecMBlxE9iGxAAV4mBCTX5Etsys9frMDnRFw== 2024-02-05T05:22:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCecMBlxE9iGxAAV4mBCTX5Etsys9frMDnRFw=="}' 2024-02-05T05:22:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58da60aa-08bb-41f5-8504-2fa1f93b3cfd -i td/osd-scrub-repair/6/new.json 2024-02-05T05:22:38.786 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:22:38.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-02-05T05:22:38.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCecMBlxE9iGxAAV4mBCTX5Etsys9frMDnRFw== --osd-uuid 58da60aa-08bb-41f5-8504-2fa1f93b3cfd 2024-02-05T05:22:38.826 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:38.824+0000 7fcfa77ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:38.879 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:38.878+0000 7fcfa77ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:38.881 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:38.879+0000 7fcfa77ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:38.881 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:38.879+0000 7fcfa77ad4c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:22:38.881 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:38.879+0000 7fcfa77ad4c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:22:38.881 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:38.879+0000 7fcfa77ad4c0 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-02-05T05:22:38.974 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:38.973+0000 7fcfa77ad4c0 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-02-05T05:22:45.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-02-05T05:22:45.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:22:45.248 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd6 key to auth repository 2024-02-05T05:22:45.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-02-05T05:22:45.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:22:45.590 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.6 2024-02-05T05:22:45.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-02-05T05:22:45.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:22:45.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:22:45.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:22:45.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:22:45.617 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:45.616+0000 7fb46bf074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:45.619 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:45.618+0000 7fb46bf074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:45.620 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:45.619+0000 7fb46bf074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:22:45.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-02-05T05:22:45.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:22:45.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-02-05T05:22:45.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:22:45.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:22:45.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:45.893 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:22:45.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:22:45.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:45.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:46.138 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:46.137+0000 7fb46bf074c0 -1 Falling back to public interface 2024-02-05T05:22:46.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:47.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:47.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:47.162 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:22:47.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:22:47.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:47.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:47.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:48.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:48.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:48.442 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:22:48.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:22:48.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:48.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:48.486 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:22:48.485+0000 7fb46bf074c0 -1 osd.6 0 log_to_monitors true 2024-02-05T05:22:48.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:49.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:49.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:49.722 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:22:49.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:22:49.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:49.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:50.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:51.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:51.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:51.096 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:22:51.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:22:51.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:51.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:51.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:52.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:52.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:52.467 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:22:52.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:22:52.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:52.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:52.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:53.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:53.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:53.749 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:22:53.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:22:53.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:53.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:54.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:55.110 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:22:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:22:55.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:55.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:55.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:56.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:56.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:56.531 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:22:56.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:22:56.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:56.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:56.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:57.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:57.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:57.864 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:22:57.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:22:57.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:57.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:58.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:22:59.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:22:59.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:22:59.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:22:59.147 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:22:59.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:22:59.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:22:59.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:00.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:00.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:00.490 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:23:00.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:23:00.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:00.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:00.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:01.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:01.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:01.984 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:23:01.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:23:01.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:01.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:02.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:03.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:03.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:03.301 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:23:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:23:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:03.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:04.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:04.754 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:23:04.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:04.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:23:04.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:04.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:05.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:06.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:06.047 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:23:06.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:06.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:23:06.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:06.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:06.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:07.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:07.604 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:23:07.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:07.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:23:07.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:07.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:08.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:09.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:09.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:09.002 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:23:09.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:23:09.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:09.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:09.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:10.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:10.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:10.590 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:23:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:23:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:11.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:12.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:12.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:12.386 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:23:12.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:23:12.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:12.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-02-05T05:23:12.752 INFO:tasks.workunit.client.0.smithi188.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3429886153,v1:127.0.0.1:6851/3429886153] [v2:127.0.0.1:6852/3429886153,v1:127.0.0.1:6853/3429886153] exists,up 58da60aa-08bb-41f5-8504-2fa1f93b3cfd 2024-02-05T05:23:12.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:23:12.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:23:12.758 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:23:12.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-02-05T05:23:12.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:23:12.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=85ce02da-bc89-475f-a06e-db80c545fa24 2024-02-05T05:23:12.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 85ce02da-bc89-475f-a06e-db80c545fa24' 2024-02-05T05:23:12.760 INFO:tasks.workunit.client.0.smithi188.stdout:add osd7 85ce02da-bc89-475f-a06e-db80c545fa24 2024-02-05T05:23:12.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:23:12.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAcMBleCILLhAAtEzrdD9WTjAon6N8haZPvg== 2024-02-05T05:23:12.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAcMBleCILLhAAtEzrdD9WTjAon6N8haZPvg=="}' 2024-02-05T05:23:12.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 85ce02da-bc89-475f-a06e-db80c545fa24 -i td/osd-scrub-repair/7/new.json 2024-02-05T05:23:13.091 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:23:13.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-02-05T05:23:13.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAcMBleCILLhAAtEzrdD9WTjAon6N8haZPvg== --osd-uuid 85ce02da-bc89-475f-a06e-db80c545fa24 2024-02-05T05:23:13.131 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:13.130+0000 7f94bde0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:13.167 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:13.166+0000 7f94bde0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:13.168 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:13.167+0000 7f94bde0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:13.168 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:13.167+0000 7f94bde0d4c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:23:13.169 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:13.167+0000 7f94bde0d4c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:23:13.169 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:13.167+0000 7f94bde0d4c0 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-02-05T05:23:13.242 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:13.240+0000 7f94bde0d4c0 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-02-05T05:23:19.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-02-05T05:23:19.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:23:19.484 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd7 key to auth repository 2024-02-05T05:23:19.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-02-05T05:23:19.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:23:19.833 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.7 2024-02-05T05:23:19.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-02-05T05:23:19.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:23:19.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:23:19.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:23:19.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:23:19.860 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:19.858+0000 7f9134c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:19.879+0000 7f9134c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:19.881 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:19.880+0000 7f9134c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:20.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-02-05T05:23:20.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:23:20.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-02-05T05:23:20.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:23:20.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:23:20.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:20.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:23:20.108 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:23:20.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:20.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:20.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:20.412 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:20.411+0000 7f9134c204c0 -1 Falling back to public interface 2024-02-05T05:23:21.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:21.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:21.380 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:23:21.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:23:21.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:21.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:21.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:22.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:22.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:22.662 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:23:22.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:23:22.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:22.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:22.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:22.772+0000 7f9134c204c0 -1 osd.7 0 log_to_monitors true 2024-02-05T05:23:22.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:23.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:23.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:23.984 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:23:23.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:23:23.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:23.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:24.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:25.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:25.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:25.344 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:23:25.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:23:25.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:25.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:25.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:26.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:26.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:26.623 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:23:26.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:23:26.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:26.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:27.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:28.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:28.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:28.065 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:23:28.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:23:28.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:28.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:28.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:29.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:29.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:29.410 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:23:29.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:23:29.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:29.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:29.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:30.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:30.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:30.819 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:23:30.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:23:30.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:30.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:31.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:32.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:32.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:32.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:23:32.099 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:23:32.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:32.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:32.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:33.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:33.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:33.426 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:23:33.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:23:33.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:33.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:33.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:34.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:34.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:34.789 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:23:34.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:23:34.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:34.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:35.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:36.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:36.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:36.070 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:23:36.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:23:36.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:36.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:36.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:37.824 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:23:37.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:23:37.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:37.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:38.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:39.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:39.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:39.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:23:39.237 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:23:39.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:39.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:39.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:40.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:40.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:40.996 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:23:40.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:23:40.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:40.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:41.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:42.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:42.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:42.525 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:23:42.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:23:42.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:42.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:42.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:43.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:43.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:43.880 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:23:43.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:23:43.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:43.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:44.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:45.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:45.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:45.714 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:23:45.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:23:45.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:45.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:46.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:47.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:47.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:47.376 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:23:47.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:23:47.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:47.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-02-05T05:23:47.652 INFO:tasks.workunit.client.0.smithi188.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1219594822,v1:127.0.0.1:6859/1219594822] [v2:127.0.0.1:6860/1219594822,v1:127.0.0.1:6861/1219594822] exists,up 85ce02da-bc89-475f-a06e-db80c545fa24 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:23:47.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:23:47.657 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:23:47.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:23:47.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:23:47.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-02-05T05:23:47.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:23:47.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ffc0984b-0a4e-49a4-b23c-6eb283db59a9 2024-02-05T05:23:47.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 ffc0984b-0a4e-49a4-b23c-6eb283db59a9' 2024-02-05T05:23:47.659 INFO:tasks.workunit.client.0.smithi188.stdout:add osd8 ffc0984b-0a4e-49a4-b23c-6eb283db59a9 2024-02-05T05:23:47.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:23:47.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDjcMBlq84NKBAAA1asLlcXvroQsWB6JqSq3A== 2024-02-05T05:23:47.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDjcMBlq84NKBAAA1asLlcXvroQsWB6JqSq3A=="}' 2024-02-05T05:23:47.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ffc0984b-0a4e-49a4-b23c-6eb283db59a9 -i td/osd-scrub-repair/8/new.json 2024-02-05T05:23:48.087 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:23:48.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-02-05T05:23:48.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjcMBlq84NKBAAA1asLlcXvroQsWB6JqSq3A== --osd-uuid ffc0984b-0a4e-49a4-b23c-6eb283db59a9 2024-02-05T05:23:48.128 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:48.127+0000 7f8601b8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:48.138 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:48.137+0000 7f8601b8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:48.139 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:48.138+0000 7f8601b8d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:48.140 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:48.138+0000 7f8601b8d4c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:23:48.140 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:48.138+0000 7f8601b8d4c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:23:48.140 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:48.138+0000 7f8601b8d4c0 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-02-05T05:23:48.247 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:48.246+0000 7f8601b8d4c0 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-02-05T05:23:54.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-02-05T05:23:54.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:23:54.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-02-05T05:23:54.186 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd8 key to auth repository 2024-02-05T05:23:54.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:23:54.534 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.8 2024-02-05T05:23:54.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-02-05T05:23:54.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:23:54.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:23:54.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:23:54.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:23:54.560 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:54.558+0000 7f20f78164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:54.562 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:54.560+0000 7f20f78164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:54.563 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:54.561+0000 7f20f78164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:23:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-02-05T05:23:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:23:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-02-05T05:23:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:23:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:23:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:23:54.807 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:23:54.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:54.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:23:55.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:55.084 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:55.082+0000 7f20f78164c0 -1 Falling back to public interface 2024-02-05T05:23:56.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:56.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:56.073 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:23:56.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:23:56.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:56.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:23:56.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:57.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:57.347 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:23:57.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:57.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:23:57.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:57.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:23:57.461 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:23:57.459+0000 7f20f78164c0 -1 osd.8 0 log_to_monitors true 2024-02-05T05:23:57.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:58.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:58.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:58.688 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:23:58.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:23:58.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:58.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:23:58.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:23:59.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:23:59.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:23:59.966 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:23:59.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:23:59.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:23:59.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:00.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:01.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:01.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:01.292 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:24:01.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:24:01.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:01.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:01.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:02.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:02.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:02.617 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:24:02.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:24:02.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:02.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:02.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:03.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:03.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:03.956 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:24:03.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:24:03.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:03.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:04.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:05.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:05.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:05.313 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:24:05.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:24:05.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:05.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:05.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:06.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:06.593 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:24:06.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:06.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:24:06.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:06.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:06.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:07.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:07.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:07.873 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:24:07.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:24:07.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:07.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:08.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:09.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:09.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:09.154 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:24:09.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:24:09.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:09.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:09.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:10.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:10.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:10.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:24:10.434 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:24:10.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:10.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:10.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:11.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:11.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:11.912 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:24:11.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:24:11.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:11.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:12.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:13.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:13.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:13.491 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:24:13.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:24:13.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:13.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:14.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:15.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:15.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:15.120 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:24:15.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:24:15.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:15.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:15.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:16.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:16.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:16.673 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:24:16.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:24:16.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:16.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:17.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:18.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:18.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:18.061 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:24:18.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:24:18.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:18.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:18.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:19.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:19.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:19.688 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:24:19.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:24:19.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:19.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:20.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:21.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:21.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:21.117 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:24:21.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:24:21.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:21.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-02-05T05:24:21.395 INFO:tasks.workunit.client.0.smithi188.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3387874355,v1:127.0.0.1:6867/3387874355] [v2:127.0.0.1:6868/3387874355,v1:127.0.0.1:6869/3387874355] exists,up ffc0984b-0a4e-49a4-b23c-6eb283db59a9 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:24:21.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:24:21.400 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:24:21.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:24:21.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:24:21.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:24:21.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:24:21.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 14c5c436-f24d-4986-ac09-ac913a12d206' 2024-02-05T05:24:21.402 INFO:tasks.workunit.client.0.smithi188.stdout:add osd9 14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:24:21.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:24:21.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFccBl0ky+GBAAhfhaC063RFGz89vDDiwDeA== 2024-02-05T05:24:21.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFccBl0ky+GBAAhfhaC063RFGz89vDDiwDeA=="}' 2024-02-05T05:24:21.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14c5c436-f24d-4986-ac09-ac913a12d206 -i td/osd-scrub-repair/9/new.json 2024-02-05T05:24:21.855 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:24:21.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-02-05T05:24:21.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFccBl0ky+GBAAhfhaC063RFGz89vDDiwDeA== --osd-uuid 14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:24:21.895 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:21.893+0000 7fb4e5b704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:21.939 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:21.938+0000 7fb4e5b704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:21.940 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:21.939+0000 7fb4e5b704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:21.941 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:21.939+0000 7fb4e5b704c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:24:21.941 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:21.939+0000 7fb4e5b704c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:24:21.941 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:21.939+0000 7fb4e5b704c0 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-02-05T05:24:22.132 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:22.131+0000 7fb4e5b704c0 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-02-05T05:24:28.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-02-05T05:24:28.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:24:28.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-02-05T05:24:28.045 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd9 key to auth repository 2024-02-05T05:24:28.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:24:28.390 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.9 2024-02-05T05:24:28.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-02-05T05:24:28.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:24:28.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:24:28.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:24:28.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:24:28.416 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:28.414+0000 7f430ab464c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:28.418 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:28.416+0000 7f430ab464c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:28.419 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:28.417+0000 7f430ab464c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:24:28.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-02-05T05:24:28.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:24:28.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:24:28.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:24:28.662 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:24:28.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:24:28.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:28.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:24:28.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:28.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:28.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:28.949 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:28.948+0000 7f430ab464c0 -1 Falling back to public interface 2024-02-05T05:24:29.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:29.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:29.929 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:24:29.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:24:29.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:29.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:30.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:31.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:31.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:31.207 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:24:31.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:24:31.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:31.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:31.293 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:24:31.292+0000 7f430ab464c0 -1 osd.9 0 log_to_monitors true 2024-02-05T05:24:31.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:32.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:32.503 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:24:32.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:32.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:24:32.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:32.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:32.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:33.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:33.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:33.781 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:24:33.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:24:33.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:33.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:34.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:35.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:35.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:35.170 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:24:35.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:24:35.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:35.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:35.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:36.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:36.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:36.448 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:24:36.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:24:36.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:36.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:36.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:37.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:37.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:37.732 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:24:37.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:24:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:38.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:39.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:39.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:39.107 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:24:39.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:24:39.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:39.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:39.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:40.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:40.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:40.387 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:24:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:24:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:40.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:41.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:41.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:41.803 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:24:41.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:24:41.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:41.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:42.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:43.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:43.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:43.145 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:24:43.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:24:43.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:43.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:43.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:44.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:44.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:44.434 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:24:44.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:24:44.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:44.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:45.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:46.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:46.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:46.410 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:24:46.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:24:46.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:46.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:47.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:48.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:48.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:48.003 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:24:48.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:24:48.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:48.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:48.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:49.312 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:24:49.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:24:49.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:49.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:49.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:50.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:50.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:50.842 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:24:50.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:24:50.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:50.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:51.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:52.200 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:24:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:24:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:52.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:53.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:53.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:53.530 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:24:53.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:24:53.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:53.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:54.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:55.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:55.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:55.362 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:24:55.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:24:55.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:55.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:55.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:24:56.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:24:56.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:24:56.682 INFO:tasks.workunit.client.0.smithi188.stdout:20 2024-02-05T05:24:56.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:24:56.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:24:56.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:24:56.962 INFO:tasks.workunit.client.0.smithi188.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2368867123,v1:127.0.0.1:6875/2368867123] [v2:127.0.0.1:6876/2368867123,v1:127.0.0.1:6877/2368867123] exists,up 14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:24:56.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:24:56.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:24:56.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:24:56.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-02-05T05:24:56.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:24:57.225 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:24:57.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:24:57.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:24:57.610 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:24:57.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:24:58.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:25:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-02-05T05:25:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:25:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:25:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:25:01.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:25:01.686 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:25:01.686 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:25:01.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:25:01.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:25:01.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:25:01.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:25:01.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:25:01.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:25:01.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:25:01.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:25:01.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:25:02.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:25:02.051 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:02.051 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:25:02.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836546 2024-02-05T05:25:02.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836546 2024-02-05T05:25:02.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546' 2024-02-05T05:25:02.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:25:02.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-02-05T05:25:02.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-02-05T05:25:02.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020' 2024-02-05T05:25:02.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:25:02.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-02-05T05:25:02.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-02-05T05:25:02.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496' 2024-02-05T05:25:02.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:25:02.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345968 2024-02-05T05:25:02.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345968 2024-02-05T05:25:02.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345968' 2024-02-05T05:25:02.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:25:02.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182441 2024-02-05T05:25:02.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182441 2024-02-05T05:25:02.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345968 4-107374182441' 2024-02-05T05:25:02.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:25:02.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018911 2024-02-05T05:25:02.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018911 2024-02-05T05:25:02.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345968 4-107374182441 5-128849018911' 2024-02-05T05:25:02.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:25:02.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855387 2024-02-05T05:25:02.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855387 2024-02-05T05:25:02.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345968 4-107374182441 5-128849018911 6-150323855387' 2024-02-05T05:25:02.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:25:02.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691859 2024-02-05T05:25:02.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691859 2024-02-05T05:25:02.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345968 4-107374182441 5-128849018911 6-150323855387 7-171798691859' 2024-02-05T05:25:02.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:25:02.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528336 2024-02-05T05:25:02.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528336 2024-02-05T05:25:02.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345968 4-107374182441 5-128849018911 6-150323855387 7-171798691859 8-193273528336' 2024-02-05T05:25:02.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:02.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:25:03.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364807 2024-02-05T05:25:03.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364807 2024-02-05T05:25:03.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836546 1-42949673020 2-64424509496 3-85899345968 4-107374182441 5-128849018911 6-150323855387 7-171798691859 8-193273528336 9-214748364807' 2024-02-05T05:25:03.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:03.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836546 2024-02-05T05:25:03.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:03.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:25:03.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836546 2024-02-05T05:25:03.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:03.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836546 2024-02-05T05:25:03.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-02-05T05:25:03.088 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836546 2024-02-05T05:25:03.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:25:03.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836546 2024-02-05T05:25:03.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:25:04.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:25:04.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:25:04.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836546 2024-02-05T05:25:04.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:04.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-02-05T05:25:04.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:04.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:25:04.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-02-05T05:25:04.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:04.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-02-05T05:25:04.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-02-05T05:25:04.639 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673020 2024-02-05T05:25:04.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:25:04.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-02-05T05:25:04.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:04.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-02-05T05:25:04.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:04.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:25:04.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-02-05T05:25:04.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:04.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-02-05T05:25:04.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-02-05T05:25:04.910 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509496 2024-02-05T05:25:04.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:25:05.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509496 -lt 64424509496 2024-02-05T05:25:05.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:05.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345968 2024-02-05T05:25:05.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:05.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:25:05.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345968 2024-02-05T05:25:05.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:05.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345968 2024-02-05T05:25:05.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2024-02-05T05:25:05.182 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345968 2024-02-05T05:25:05.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:25:05.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345968 2024-02-05T05:25:05.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:05.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182441 2024-02-05T05:25:05.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:05.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:25:05.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182441 2024-02-05T05:25:05.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:05.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182441 2024-02-05T05:25:05.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182441' 2024-02-05T05:25:05.452 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.4 seq 107374182441 2024-02-05T05:25:05.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:25:05.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182441 -lt 107374182441 2024-02-05T05:25:05.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:05.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018911 2024-02-05T05:25:05.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:05.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:25:05.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018911 2024-02-05T05:25:05.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:05.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018911 2024-02-05T05:25:05.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018911' 2024-02-05T05:25:05.722 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.5 seq 128849018911 2024-02-05T05:25:05.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:25:05.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018912 -lt 128849018911 2024-02-05T05:25:05.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:05.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855387 2024-02-05T05:25:05.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:05.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:25:05.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855387 2024-02-05T05:25:05.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:05.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855387 2024-02-05T05:25:05.997 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.6 seq 150323855387 2024-02-05T05:25:05.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855387' 2024-02-05T05:25:05.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:25:06.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855388 -lt 150323855387 2024-02-05T05:25:06.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:06.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691859 2024-02-05T05:25:06.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:06.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:25:06.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691859 2024-02-05T05:25:06.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:06.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691859 2024-02-05T05:25:06.269 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.7 seq 171798691859 2024-02-05T05:25:06.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691859' 2024-02-05T05:25:06.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:25:06.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691859 -lt 171798691859 2024-02-05T05:25:06.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:06.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528336 2024-02-05T05:25:06.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:06.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:25:06.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528336 2024-02-05T05:25:06.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:06.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528336 2024-02-05T05:25:06.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2024-02-05T05:25:06.540 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.8 seq 193273528336 2024-02-05T05:25:06.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:25:06.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528336 -lt 193273528336 2024-02-05T05:25:06.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:06.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364807 2024-02-05T05:25:06.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:06.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:25:06.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364807 2024-02-05T05:25:06.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:06.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364807 2024-02-05T05:25:06.811 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.9 seq 214748364807 2024-02-05T05:25:06.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364807' 2024-02-05T05:25:06.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:25:07.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364807 -lt 214748364807 2024-02-05T05:25:07.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:25:07.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:07.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:07.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:25:07.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:25:07.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:25:07.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:25:07.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:25:07.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:25:07.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:25:07.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:25:07.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:25:07.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:25:07.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:07.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:08.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:25:08.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:25:08.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:25:08.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-02-05T05:25:08.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:25:08.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:25:08.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:25:08.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:25:08.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-02-05T05:25:08.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:25:08.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:25:09.410 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:25:09.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:25:10.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:25:10.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:25:10.939 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T05:25:10.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:25:10.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:25:10.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:25:10.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:25:10.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:25:10.952 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:25:10.952 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:25:10.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:25:10.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:25:10.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:25:11.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:25:11.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:25:11.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:25:11.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:25:11.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:25:11.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:25:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:25:11.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:25:11.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:11.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:25:11.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-02-05T05:25:11.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-02-05T05:25:11.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-02-05T05:25:11.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:11.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:25:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673023 2024-02-05T05:25:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673023 2024-02-05T05:25:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023' 2024-02-05T05:25:11.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:11.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:25:11.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-02-05T05:25:11.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-02-05T05:25:11.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498' 2024-02-05T05:25:11.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:11.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:25:11.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345971 2024-02-05T05:25:11.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345971 2024-02-05T05:25:11.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498 3-85899345971' 2024-02-05T05:25:11.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:11.926 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:25:12.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182444 2024-02-05T05:25:12.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182444 2024-02-05T05:25:12.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498 3-85899345971 4-107374182444' 2024-02-05T05:25:12.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:12.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:25:12.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018914 2024-02-05T05:25:12.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018914 2024-02-05T05:25:12.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498 3-85899345971 4-107374182444 5-128849018914' 2024-02-05T05:25:12.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:12.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:25:12.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-02-05T05:25:12.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-02-05T05:25:12.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498 3-85899345971 4-107374182444 5-128849018914 6-150323855390' 2024-02-05T05:25:12.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:12.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:25:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691862 2024-02-05T05:25:12.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691862 2024-02-05T05:25:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498 3-85899345971 4-107374182444 5-128849018914 6-150323855390 7-171798691862' 2024-02-05T05:25:12.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:12.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:25:12.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528339 2024-02-05T05:25:12.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528339 2024-02-05T05:25:12.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498 3-85899345971 4-107374182444 5-128849018914 6-150323855390 7-171798691862 8-193273528339' 2024-02-05T05:25:12.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:12.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:25:12.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364810 2024-02-05T05:25:12.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364810 2024-02-05T05:25:12.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673023 2-64424509498 3-85899345971 4-107374182444 5-128849018914 6-150323855390 7-171798691862 8-193273528339 9-214748364810' 2024-02-05T05:25:12.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:12.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-02-05T05:25:12.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:12.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:25:12.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-02-05T05:25:12.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:12.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-02-05T05:25:12.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-02-05T05:25:12.548 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836549 2024-02-05T05:25:12.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:25:12.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-02-05T05:25:12.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:12.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673023 2024-02-05T05:25:12.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:12.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:25:12.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673023 2024-02-05T05:25:12.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:12.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673023 2024-02-05T05:25:12.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-02-05T05:25:12.819 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673023 2024-02-05T05:25:12.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:25:13.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673023 2024-02-05T05:25:13.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:13.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-02-05T05:25:13.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:13.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:25:13.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-02-05T05:25:13.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:13.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-02-05T05:25:13.091 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509498 2024-02-05T05:25:13.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-02-05T05:25:13.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:25:13.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509497 -lt 64424509498 2024-02-05T05:25:13.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:25:14.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:25:14.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:25:14.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509499 -lt 64424509498 2024-02-05T05:25:14.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:14.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345971 2024-02-05T05:25:14.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:14.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:25:14.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345971 2024-02-05T05:25:14.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:14.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345971 2024-02-05T05:25:14.653 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345971 2024-02-05T05:25:14.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345971' 2024-02-05T05:25:14.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:25:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345971 -lt 85899345971 2024-02-05T05:25:14.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:14.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182444 2024-02-05T05:25:14.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:25:14.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182444 2024-02-05T05:25:14.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:14.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182444 2024-02-05T05:25:14.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2024-02-05T05:25:14.924 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.4 seq 107374182444 2024-02-05T05:25:14.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:25:15.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182444 -lt 107374182444 2024-02-05T05:25:15.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:15.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018914 2024-02-05T05:25:15.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:15.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:25:15.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018914 2024-02-05T05:25:15.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:15.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018914 2024-02-05T05:25:15.194 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.5 seq 128849018914 2024-02-05T05:25:15.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018914' 2024-02-05T05:25:15.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:25:15.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018915 -lt 128849018914 2024-02-05T05:25:15.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:15.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-02-05T05:25:15.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:15.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:25:15.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-02-05T05:25:15.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:15.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-02-05T05:25:15.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-02-05T05:25:15.465 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.6 seq 150323855390 2024-02-05T05:25:15.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:25:15.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855391 -lt 150323855390 2024-02-05T05:25:15.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:15.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691862 2024-02-05T05:25:15.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:15.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:25:15.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691862 2024-02-05T05:25:15.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691862 2024-02-05T05:25:15.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691862' 2024-02-05T05:25:15.737 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.7 seq 171798691862 2024-02-05T05:25:15.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:25:16.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691862 2024-02-05T05:25:16.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:16.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528339 2024-02-05T05:25:16.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:16.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:25:16.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528339 2024-02-05T05:25:16.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:16.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528339 2024-02-05T05:25:16.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-02-05T05:25:16.012 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.8 seq 193273528339 2024-02-05T05:25:16.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:25:16.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528339 -lt 193273528339 2024-02-05T05:25:16.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:16.278 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364810 2024-02-05T05:25:16.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:16.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:25:16.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364810 2024-02-05T05:25:16.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:16.281 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.9 seq 214748364810 2024-02-05T05:25:16.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364810 2024-02-05T05:25:16.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2024-02-05T05:25:16.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:25:16.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364810 -lt 214748364810 2024-02-05T05:25:16.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:25:16.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:16.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:16.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:25:16.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:25:16.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:25:16.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:25:16.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:25:16.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:25:16.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:25:16.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:25:17.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:25:17.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:25:17.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:17.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:17.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:25:17.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:25:17.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:25:17.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:25:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:25:17.982 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:25:17.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:25:18.494 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:25:18.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:25:18.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:25:18.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:25:18.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T05:25:18.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T05:25:18.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:25:18.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:25:18.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:25:19.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-02-05T05:25:19.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T05:25:19.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:25:19.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T05:25:19.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-02-05T05:25:19.247 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:25:19.247 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr:9 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:19.520 INFO:tasks.workunit.client.0.smithi188.stderr:7' 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:25:19.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:25:19.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:25:19.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:25:19.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:25:19.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:25:19.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:25:19.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:25:19.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:25:19.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:25:19.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:25:21.642 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:25:22.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:25:22.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:25:22.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:25:22.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:25:22.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:25:22.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:25:22.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:25:22.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:25:22.166 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.3 2024-02-05T05:25:22.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:25:22.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:25:22.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:25:22.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:25:22.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:25:22.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:25:22.192 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:22.191+0000 7f7d099374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:22.194 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:22.193+0000 7f7d099374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:22.196 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:22.194+0000 7f7d099374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:22.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:22.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:25:22.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:22.724 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:22.723+0000 7f7d099374c0 -1 Falling back to public interface 2024-02-05T05:25:23.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:23.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:23.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:25:23.713 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:23.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:23.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:25:24.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:25.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:25.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:25.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:25:25.012 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:25.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:25.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:25:25.102 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:25.101+0000 7f7d099374c0 -1 osd.3 68 log_to_monitors true 2024-02-05T05:25:25.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:26.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:26.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:26.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:25:26.309 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:26.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:26.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [20,68) [v2:127.0.0.1:6826/2140993807,v1:127.0.0.1:6827/2140993807] [v2:127.0.0.1:6828/2140993807,v1:127.0.0.1:6829/2140993807] exists,up f68a09e8-d5b4-4da0-8457-46288076ddeb 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:25:26.719 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:25:26.720 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:25:26.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:25:26.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:25:26.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:25:26.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:25:26.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:25:26.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:25:26.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:25:26.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:25:26.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:25:27.200 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:25:27.201 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:25:27.201 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:25:27.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:25:27.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:27.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:25:27.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-02-05T05:25:27.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-02-05T05:25:27.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-02-05T05:25:27.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:27.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:25:27.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673027 2024-02-05T05:25:27.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673027 2024-02-05T05:25:27.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027' 2024-02-05T05:25:27.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:27.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:25:27.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-02-05T05:25:27.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-02-05T05:25:27.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503' 2024-02-05T05:25:27.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:27.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:25:27.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-02-05T05:25:27.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-02-05T05:25:27.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314' 2024-02-05T05:25:27.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:27.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:25:27.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-02-05T05:25:27.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-02-05T05:25:27.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182448' 2024-02-05T05:25:27.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:27.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:25:27.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018918 2024-02-05T05:25:27.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018918 2024-02-05T05:25:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182448 5-128849018918' 2024-02-05T05:25:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:27.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:25:28.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855395 2024-02-05T05:25:28.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855395 2024-02-05T05:25:28.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182448 5-128849018918 6-150323855395' 2024-02-05T05:25:28.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:28.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:25:28.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691866 2024-02-05T05:25:28.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691866 2024-02-05T05:25:28.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182448 5-128849018918 6-150323855395 7-171798691866' 2024-02-05T05:25:28.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:28.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:25:28.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528343 2024-02-05T05:25:28.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528343 2024-02-05T05:25:28.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182448 5-128849018918 6-150323855395 7-171798691866 8-193273528343' 2024-02-05T05:25:28.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:28.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:25:28.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364814 2024-02-05T05:25:28.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364814 2024-02-05T05:25:28.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673027 2-64424509503 3-309237645314 4-107374182448 5-128849018918 6-150323855395 7-171798691866 8-193273528343 9-214748364814' 2024-02-05T05:25:28.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:28.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-02-05T05:25:28.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:28.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:25:28.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-02-05T05:25:28.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:28.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-02-05T05:25:28.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-02-05T05:25:28.355 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836553 2024-02-05T05:25:28.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:25:28.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836553 -lt 21474836553 2024-02-05T05:25:28.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:28.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673027 2024-02-05T05:25:28.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:28.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:25:28.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673027 2024-02-05T05:25:28.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:28.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673027 2024-02-05T05:25:28.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673027' 2024-02-05T05:25:28.623 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673027 2024-02-05T05:25:28.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:25:28.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673027 -lt 42949673027 2024-02-05T05:25:28.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:28.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-02-05T05:25:28.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:28.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:25:28.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-02-05T05:25:28.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:28.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-02-05T05:25:28.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-02-05T05:25:28.895 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509503 2024-02-05T05:25:28.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:25:29.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-02-05T05:25:29.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:29.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645314 2024-02-05T05:25:29.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:29.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:25:29.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645314 2024-02-05T05:25:29.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:29.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-02-05T05:25:29.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645314' 2024-02-05T05:25:29.193 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 309237645314 2024-02-05T05:25:29.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:25:29.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-02-05T05:25:29.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:29.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-02-05T05:25:29.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:29.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:25:29.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-02-05T05:25:29.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:29.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-02-05T05:25:29.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-02-05T05:25:29.466 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182448 2024-02-05T05:25:29.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:25:29.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182447 -lt 107374182448 2024-02-05T05:25:29.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:25:30.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:25:30.738 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:25:31.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182448 -lt 107374182448 2024-02-05T05:25:31.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:31.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018918 2024-02-05T05:25:31.016 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:31.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:25:31.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018918 2024-02-05T05:25:31.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:31.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018918 2024-02-05T05:25:31.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018918' 2024-02-05T05:25:31.019 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 128849018918 2024-02-05T05:25:31.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:25:31.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018919 -lt 128849018918 2024-02-05T05:25:31.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:31.285 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855395 2024-02-05T05:25:31.285 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:31.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:25:31.287 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855395 2024-02-05T05:25:31.287 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:31.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855395 2024-02-05T05:25:31.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2024-02-05T05:25:31.288 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855395 2024-02-05T05:25:31.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:25:31.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855395 -lt 150323855395 2024-02-05T05:25:31.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:31.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691866 2024-02-05T05:25:31.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:31.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:25:31.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691866 2024-02-05T05:25:31.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691866 2024-02-05T05:25:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691866' 2024-02-05T05:25:31.560 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691866 2024-02-05T05:25:31.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:25:31.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691866 -lt 171798691866 2024-02-05T05:25:31.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:31.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528343 2024-02-05T05:25:31.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:31.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:25:31.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528343 2024-02-05T05:25:31.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:31.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528343 2024-02-05T05:25:31.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528343' 2024-02-05T05:25:31.834 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528343 2024-02-05T05:25:31.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:25:32.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528343 -lt 193273528343 2024-02-05T05:25:32.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:32.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364814 2024-02-05T05:25:32.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:32.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:25:32.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364814 2024-02-05T05:25:32.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:32.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364814 2024-02-05T05:25:32.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364814' 2024-02-05T05:25:32.127 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 214748364814 2024-02-05T05:25:32.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:25:32.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364814 -lt 214748364814 2024-02-05T05:25:32.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:25:32.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:32.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:32.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:25:32.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:25:32.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:25:32.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:25:32.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:25:32.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:25:32.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:25:32.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:25:32.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:25:32.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:25:32.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:32.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:33.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:25:33.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:25:33.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:25:33.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:25:33.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:25:33.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:25:33.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:25:33.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:25:33.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:25:33.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:25:33.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:25:33.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:25:33.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:25:33.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:25:33.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:25:33.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:25:33.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:25:09.048019+0000 2024-02-05T05:25:33.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:25:34.114 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:25:09.048019+0000 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:25:09.048019+0000 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:25:34.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:25:34.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:25:34.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:25:34.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:25:09.048019+0000 '>' 2024-02-05T05:25:09.048019+0000 2024-02-05T05:25:34.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:25:35.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:25:35.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:25:35.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:25:35.392 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:25:35.392 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:25:35.392 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:25:35.392 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:25:35.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:25:09.048019+0000 '>' 2024-02-05T05:25:09.048019+0000 2024-02-05T05:25:35.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:25:36.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:25:36.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:25:36.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:25:36.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:25:36.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:25:36.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:25:36.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:25:35.168699+0000 '>' 2024-02-05T05:25:09.048019+0000 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:25:36.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:25:36.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:25:37.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:25:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:25:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:25:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:25:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:25:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:25:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:25:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:25:39.077 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:25:39.077 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:25:39.077 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:25:39.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:25:39.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:25:39.357 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:25:39.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:25:39.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:25:39.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:25:39.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:25:39.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:25:39.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:25:39.359 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.9 2024-02-05T05:25:39.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:25:39.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:25:39.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:25:39.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:25:39.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:25:39.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:25:39.386 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:39.384+0000 7ff54541a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:39.388 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:39.387+0000 7ff54541a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:39.389 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:39.388+0000 7ff54541a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:39.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:25:39.907 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:39.906+0000 7ff54541a4c0 -1 Falling back to public interface 2024-02-05T05:25:40.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:41.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:41.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:41.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:25:41.025 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:41.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:41.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:25:41.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:42.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:42.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:42.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:25:42.302 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:42.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:42.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:25:42.332 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:42.330+0000 7ff54541a4c0 -1 osd.9 73 log_to_monitors true 2024-02-05T05:25:42.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:43.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:43.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:43.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:25:43.580 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:43.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:43.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:osd.9 up in weight 1 up_from 77 up_thru 0 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6874/2017736704,v1:127.0.0.1:6875/2017736704] [v2:127.0.0.1:6876/2017736704,v1:127.0.0.1:6877/2017736704] exists,up 14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:25:43.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:25:43.901 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:25:43.901 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:25:43.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:25:43.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:25:43.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:25:43.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:25:43.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:25:43.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:25:43.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:25:43.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:25:43.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:25:44.469 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:25:44.470 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:25:44.470 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:25:44.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:25:44.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:44.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:25:44.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-02-05T05:25:44.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-02-05T05:25:44.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-02-05T05:25:44.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:44.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:25:44.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673032 2024-02-05T05:25:44.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673032 2024-02-05T05:25:44.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032' 2024-02-05T05:25:44.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:44.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:25:44.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-02-05T05:25:44.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-02-05T05:25:44.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507' 2024-02-05T05:25:44.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:44.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:25:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645318 2024-02-05T05:25:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645318 2024-02-05T05:25:44.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507 3-309237645318' 2024-02-05T05:25:44.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:44.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:25:45.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182452 2024-02-05T05:25:45.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182452 2024-02-05T05:25:45.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507 3-309237645318 4-107374182452' 2024-02-05T05:25:45.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:45.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:25:45.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018923 2024-02-05T05:25:45.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018923 2024-02-05T05:25:45.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507 3-309237645318 4-107374182452 5-128849018923' 2024-02-05T05:25:45.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:45.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:25:45.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855399 2024-02-05T05:25:45.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855399 2024-02-05T05:25:45.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507 3-309237645318 4-107374182452 5-128849018923 6-150323855399' 2024-02-05T05:25:45.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:45.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:25:45.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691870 2024-02-05T05:25:45.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691870 2024-02-05T05:25:45.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507 3-309237645318 4-107374182452 5-128849018923 6-150323855399 7-171798691870' 2024-02-05T05:25:45.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:45.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:25:45.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528347 2024-02-05T05:25:45.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528347 2024-02-05T05:25:45.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507 3-309237645318 4-107374182452 5-128849018923 6-150323855399 7-171798691870 8-193273528347' 2024-02-05T05:25:45.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:45.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:25:45.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-02-05T05:25:45.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-02-05T05:25:45.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673032 2-64424509507 3-309237645318 4-107374182452 5-128849018923 6-150323855399 7-171798691870 8-193273528347 9-330712481794' 2024-02-05T05:25:45.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:45.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-02-05T05:25:45.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:45.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:25:45.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-02-05T05:25:45.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:45.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-02-05T05:25:45.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-02-05T05:25:45.595 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836557 2024-02-05T05:25:45.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:25:45.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836557 2024-02-05T05:25:45.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:45.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673032 2024-02-05T05:25:45.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:45.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:25:45.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673032 2024-02-05T05:25:45.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:45.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673032 2024-02-05T05:25:45.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673032' 2024-02-05T05:25:45.866 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673032 2024-02-05T05:25:45.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:25:46.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673032 -lt 42949673032 2024-02-05T05:25:46.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:46.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-02-05T05:25:46.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:46.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:25:46.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-02-05T05:25:46.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:46.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-02-05T05:25:46.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-02-05T05:25:46.138 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509507 2024-02-05T05:25:46.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:25:46.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509507 2024-02-05T05:25:46.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:46.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645318 2024-02-05T05:25:46.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:46.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:25:46.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645318 2024-02-05T05:25:46.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:46.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645318 2024-02-05T05:25:46.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645318' 2024-02-05T05:25:46.410 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 309237645318 2024-02-05T05:25:46.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:25:46.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645319 -lt 309237645318 2024-02-05T05:25:46.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:46.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182452 2024-02-05T05:25:46.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:46.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:25:46.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182452 2024-02-05T05:25:46.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:46.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182452 2024-02-05T05:25:46.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-02-05T05:25:46.683 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182452 2024-02-05T05:25:46.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:25:46.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182453 -lt 107374182452 2024-02-05T05:25:46.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:46.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018923 2024-02-05T05:25:46.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:46.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:25:46.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018923 2024-02-05T05:25:46.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:46.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018923 2024-02-05T05:25:46.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018923' 2024-02-05T05:25:46.955 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 128849018923 2024-02-05T05:25:46.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:25:47.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018923 -lt 128849018923 2024-02-05T05:25:47.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:47.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855399 2024-02-05T05:25:47.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:47.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:25:47.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855399 2024-02-05T05:25:47.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:47.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855399 2024-02-05T05:25:47.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855399' 2024-02-05T05:25:47.229 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855399 2024-02-05T05:25:47.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:25:47.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855399 2024-02-05T05:25:47.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:47.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691870 2024-02-05T05:25:47.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:47.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:25:47.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691870 2024-02-05T05:25:47.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:47.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691870 2024-02-05T05:25:47.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-02-05T05:25:47.502 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691870 2024-02-05T05:25:47.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:25:47.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691871 -lt 171798691870 2024-02-05T05:25:47.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:47.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528347 2024-02-05T05:25:47.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:47.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:25:47.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528347 2024-02-05T05:25:47.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:47.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528347 2024-02-05T05:25:47.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2024-02-05T05:25:47.774 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528347 2024-02-05T05:25:47.774 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:25:48.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528348 -lt 193273528347 2024-02-05T05:25:48.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:48.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481794 2024-02-05T05:25:48.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:48.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:25:48.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481794 2024-02-05T05:25:48.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:48.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-02-05T05:25:48.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481794' 2024-02-05T05:25:48.056 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 330712481794 2024-02-05T05:25:48.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:25:48.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481795 -lt 330712481794 2024-02-05T05:25:48.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:25:48.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:48.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:25:48.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:25:48.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:25:48.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:25:48.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:25:48.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:25:49.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:25:49.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:25:49.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:25:49.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:25:49.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:25:49.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:25:49.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:25:49.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:25:49.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:25:49.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:25:49.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:25:49.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:25:49.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:25:49.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:25:49.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:25:51.678 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:25:52.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:25:52.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:25:52.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:25:52.202 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.5 2024-02-05T05:25:52.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:25:52.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:25:52.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:25:52.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:25:52.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:25:52.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:25:52.229 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:52.227+0000 7fec6d9ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:52.231 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:52.230+0000 7fec6d9ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:52.232 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:52.231+0000 7fec6d9ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:25:52.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:25:52.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:25:52.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:25:52.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:25:52.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:25:52.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:52.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:25:52.481 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:25:52.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:52.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:25:52.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:52.768 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:52.767+0000 7fec6d9ad4c0 -1 Falling back to public interface 2024-02-05T05:25:53.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:53.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:53.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:25:53.752 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:53.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:53.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:25:54.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:55.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:55.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:55.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:25:55.053 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:55.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:55.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:25:55.134 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:25:55.132+0000 7fec6d9ad4c0 -1 osd.5 79 log_to_monitors true 2024-02-05T05:25:55.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:25:56.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:25:56.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:25:56.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:25:56.442 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:56.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:25:56.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:25:56.725 INFO:tasks.workunit.client.0.smithi188.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/83271969,v1:127.0.0.1:6843/83271969] [v2:127.0.0.1:6844/83271969,v1:127.0.0.1:6845/83271969] exists,up 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:25:56.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:25:56.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:25:56.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:25:56.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:25:56.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:25:56.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:25:56.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:25:56.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:25:57.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:57.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:25:57.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-02-05T05:25:57.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-02-05T05:25:57.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-02-05T05:25:57.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:57.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:25:57.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-02-05T05:25:57.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-02-05T05:25:57.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035' 2024-02-05T05:25:57.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:57.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:25:57.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-02-05T05:25:57.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-02-05T05:25:57.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511' 2024-02-05T05:25:57.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:57.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:25:57.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645322 2024-02-05T05:25:57.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645322 2024-02-05T05:25:57.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511 3-309237645322' 2024-02-05T05:25:57.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:57.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:25:57.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182456 2024-02-05T05:25:57.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182456 2024-02-05T05:25:57.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511 3-309237645322 4-107374182456' 2024-02-05T05:25:57.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:57.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:25:58.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-02-05T05:25:58.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-02-05T05:25:58.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511 3-309237645322 4-107374182456 5-360777252866' 2024-02-05T05:25:58.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:58.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:25:58.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855403 2024-02-05T05:25:58.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855403 2024-02-05T05:25:58.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511 3-309237645322 4-107374182456 5-360777252866 6-150323855403' 2024-02-05T05:25:58.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:58.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:25:58.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691874 2024-02-05T05:25:58.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691874 2024-02-05T05:25:58.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511 3-309237645322 4-107374182456 5-360777252866 6-150323855403 7-171798691874' 2024-02-05T05:25:58.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:58.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:25:58.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528351 2024-02-05T05:25:58.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528351 2024-02-05T05:25:58.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511 3-309237645322 4-107374182456 5-360777252866 6-150323855403 7-171798691874 8-193273528351' 2024-02-05T05:25:58.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:25:58.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:25:58.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481798 2024-02-05T05:25:58.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481798 2024-02-05T05:25:58.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673035 2-64424509511 3-309237645322 4-107374182456 5-360777252866 6-150323855403 7-171798691874 8-193273528351 9-330712481798' 2024-02-05T05:25:58.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:58.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-02-05T05:25:58.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:58.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:25:58.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-02-05T05:25:58.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:58.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-02-05T05:25:58.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-02-05T05:25:58.459 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836561 2024-02-05T05:25:58.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:25:58.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836561 -lt 21474836561 2024-02-05T05:25:58.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:58.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-02-05T05:25:58.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:58.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:25:58.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-02-05T05:25:58.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:58.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-02-05T05:25:58.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-02-05T05:25:58.730 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673035 2024-02-05T05:25:58.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:25:59.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673035 2024-02-05T05:25:59.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:25:59.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-02-05T05:25:59.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:25:59.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:25:59.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-02-05T05:25:59.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:25:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-02-05T05:25:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-02-05T05:25:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509511 2024-02-05T05:25:59.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:25:59.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509511 2024-02-05T05:25:59.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:26:00.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:26:00.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:26:00.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509511 2024-02-05T05:26:00.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:00.556 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645322 2024-02-05T05:26:00.556 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:00.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:26:00.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645322 2024-02-05T05:26:00.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:00.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645322 2024-02-05T05:26:00.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645322' 2024-02-05T05:26:00.558 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 309237645322 2024-02-05T05:26:00.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:26:00.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645322 -lt 309237645322 2024-02-05T05:26:00.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:00.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182456 2024-02-05T05:26:00.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:00.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:26:00.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182456 2024-02-05T05:26:00.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:00.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182456 2024-02-05T05:26:00.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182456' 2024-02-05T05:26:00.832 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182456 2024-02-05T05:26:00.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:26:01.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182456 -lt 107374182456 2024-02-05T05:26:01.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:01.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-02-05T05:26:01.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:01.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:26:01.104 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-02-05T05:26:01.104 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:01.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-02-05T05:26:01.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-02-05T05:26:01.105 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 360777252866 2024-02-05T05:26:01.105 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:26:01.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-02-05T05:26:01.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:01.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855403 2024-02-05T05:26:01.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:01.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:26:01.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855403 2024-02-05T05:26:01.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:01.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855403 2024-02-05T05:26:01.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-02-05T05:26:01.374 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855403 2024-02-05T05:26:01.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:26:01.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855403 -lt 150323855403 2024-02-05T05:26:01.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:01.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691874 2024-02-05T05:26:01.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:01.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:26:01.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691874 2024-02-05T05:26:01.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:01.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691874 2024-02-05T05:26:01.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2024-02-05T05:26:01.646 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691874 2024-02-05T05:26:01.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:26:01.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691874 2024-02-05T05:26:01.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:01.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528351 2024-02-05T05:26:01.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:26:01.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528351 2024-02-05T05:26:01.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:01.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528351 2024-02-05T05:26:01.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2024-02-05T05:26:01.947 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528351 2024-02-05T05:26:01.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:26:02.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528351 -lt 193273528351 2024-02-05T05:26:02.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:02.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-330712481798 2024-02-05T05:26:02.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:02.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:26:02.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-330712481798 2024-02-05T05:26:02.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:02.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481798 2024-02-05T05:26:02.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 330712481798' 2024-02-05T05:26:02.219 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 330712481798 2024-02-05T05:26:02.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:26:02.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481798 -lt 330712481798 2024-02-05T05:26:02.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:26:02.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:02.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:02.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:26:02.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:26:02.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:26:02.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:26:02.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:26:02.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:26:02.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:26:02.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:26:03.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:26:03.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:26:03.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:03.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:03.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:26:03.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:26:03.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:26:03.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-02-05T05:26:03.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:26:03.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:26:03.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:26:03.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:26:03.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-02-05T05:26:03.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-02-05T05:26:03.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:26:03.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:26:03.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:03.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:03.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:03.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:03.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:25:35.168699+0000 2024-02-05T05:26:03.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:26:04.252 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:26:04.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:25:35.168699+0000 2024-02-05T05:26:04.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:26:04.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:25:35.168699+0000 2024-02-05T05:26:04.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:26:04.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:26:04.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:26:04.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:26:04.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:04.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:04.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:04.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:04.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:25:35.168699+0000 '>' 2024-02-05T05:25:35.168699+0000 2024-02-05T05:26:04.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:26:05.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:26:05.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:26:05.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:26:05.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:05.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:05.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:05.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:05.153204+0000 '>' 2024-02-05T05:25:35.168699+0000 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:26:05.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:26:05.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:26:05.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:26:05.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:26:05.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:26:05.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:26:06.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:26:06.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:26:06.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:26:06.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:26:06.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:26:06.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:26:06.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:26:06.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:26:08.048 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:26:08.048 INFO:tasks.workunit.client.0.smithi188.stdout:hinfo_key 2024-02-05T05:26:08.048 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:26:08.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:26:08.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:26:08.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:26:08.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:26:08.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:26:08.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:26:08.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:26:08.323 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.9 2024-02-05T05:26:08.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:26:08.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:26:08.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:26:08.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:26:08.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:26:08.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:26:08.349 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:26:08.348+0000 7f96a81074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:08.351 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:26:08.350+0000 7f96a81074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:08.352 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:26:08.351+0000 7f96a81074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:08.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:08.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:26:08.884+0000 7f96a81074c0 -1 Falling back to public interface 2024-02-05T05:26:09.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:09.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:09.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:26:09.870 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:26:09.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:09.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:10.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:11.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:11.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:11.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:26:11.150 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:26:11.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:11.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:11.163 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:26:11.161+0000 7f96a81074c0 -1 osd.9 86 log_to_monitors true 2024-02-05T05:26:11.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:12.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:12.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:12.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:26:12.489 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:26:12.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:12.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [77,86) [v2:127.0.0.1:6874/1194363284,v1:127.0.0.1:6875/1194363284] [v2:127.0.0.1:6876/1194363284,v1:127.0.0.1:6877/1194363284] exists,up 14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:26:13.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:26:13.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:26:13.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:26:13.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:26:13.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:26:13.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:26:13.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:26:13.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:26:13.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:26:13.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:26:13.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:26:13.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:4 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:5 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:6 2024-02-05T05:26:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:7 2024-02-05T05:26:13.580 INFO:tasks.workunit.client.0.smithi188.stderr:8 2024-02-05T05:26:13.580 INFO:tasks.workunit.client.0.smithi188.stderr:9' 2024-02-05T05:26:13.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:26:13.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:13.580 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:26:13.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-02-05T05:26:13.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-02-05T05:26:13.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-02-05T05:26:13.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:13.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:26:13.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673039 2024-02-05T05:26:13.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673039 2024-02-05T05:26:13.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039' 2024-02-05T05:26:13.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:13.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:26:14.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-02-05T05:26:14.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-02-05T05:26:14.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515' 2024-02-05T05:26:14.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:14.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:26:14.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645326 2024-02-05T05:26:14.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645326 2024-02-05T05:26:14.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515 3-309237645326' 2024-02-05T05:26:14.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:14.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:26:14.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182460 2024-02-05T05:26:14.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182460 2024-02-05T05:26:14.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515 3-309237645326 4-107374182460' 2024-02-05T05:26:14.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:14.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:26:14.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252870 2024-02-05T05:26:14.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252870 2024-02-05T05:26:14.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515 3-309237645326 4-107374182460 5-360777252870' 2024-02-05T05:26:14.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:14.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:26:14.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855407 2024-02-05T05:26:14.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855407 2024-02-05T05:26:14.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515 3-309237645326 4-107374182460 5-360777252870 6-150323855407' 2024-02-05T05:26:14.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:14.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:26:14.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691878 2024-02-05T05:26:14.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691878 2024-02-05T05:26:14.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515 3-309237645326 4-107374182460 5-360777252870 6-150323855407 7-171798691878' 2024-02-05T05:26:14.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:14.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:26:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528355 2024-02-05T05:26:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528355 2024-02-05T05:26:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515 3-309237645326 4-107374182460 5-360777252870 6-150323855407 7-171798691878 8-193273528355' 2024-02-05T05:26:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:14.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:26:14.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-02-05T05:26:14.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-02-05T05:26:14.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673039 2-64424509515 3-309237645326 4-107374182460 5-360777252870 6-150323855407 7-171798691878 8-193273528355 9-386547056642' 2024-02-05T05:26:14.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:14.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-02-05T05:26:14.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:14.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:26:14.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-02-05T05:26:14.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:14.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-02-05T05:26:14.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-02-05T05:26:14.764 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836565 2024-02-05T05:26:14.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:15.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836565 2024-02-05T05:26:15.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:26:16.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:26:16.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:16.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836566 -lt 21474836565 2024-02-05T05:26:16.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:16.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673039 2024-02-05T05:26:16.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:16.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:26:16.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673039 2024-02-05T05:26:16.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:16.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673039 2024-02-05T05:26:16.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-02-05T05:26:16.316 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949673039 2024-02-05T05:26:16.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:26:16.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673039 2024-02-05T05:26:16.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:16.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-02-05T05:26:16.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:16.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:26:16.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-02-05T05:26:16.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:16.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-02-05T05:26:16.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-02-05T05:26:16.590 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509515 2024-02-05T05:26:16.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:26:16.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509515 2024-02-05T05:26:16.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:16.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645326 2024-02-05T05:26:16.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:26:16.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645326 2024-02-05T05:26:16.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:16.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645326 2024-02-05T05:26:16.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645326' 2024-02-05T05:26:16.864 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.3 seq 309237645326 2024-02-05T05:26:16.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:26:17.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645327 -lt 309237645326 2024-02-05T05:26:17.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:17.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182460 2024-02-05T05:26:17.134 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:17.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:26:17.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182460 2024-02-05T05:26:17.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:17.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182460 2024-02-05T05:26:17.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-02-05T05:26:17.137 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.4 seq 107374182460 2024-02-05T05:26:17.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:26:17.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182461 -lt 107374182460 2024-02-05T05:26:17.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:17.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252870 2024-02-05T05:26:17.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:17.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:26:17.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252870 2024-02-05T05:26:17.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:17.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252870 2024-02-05T05:26:17.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252870' 2024-02-05T05:26:17.407 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.5 seq 360777252870 2024-02-05T05:26:17.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:26:17.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252871 -lt 360777252870 2024-02-05T05:26:17.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:17.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855407 2024-02-05T05:26:17.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:17.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:26:17.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855407 2024-02-05T05:26:17.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:17.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855407 2024-02-05T05:26:17.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855407' 2024-02-05T05:26:17.680 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.6 seq 150323855407 2024-02-05T05:26:17.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:26:18.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855407 -lt 150323855407 2024-02-05T05:26:18.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:18.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691878 2024-02-05T05:26:18.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:18.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:26:18.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691878 2024-02-05T05:26:18.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691878 2024-02-05T05:26:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-02-05T05:26:18.052 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.7 seq 171798691878 2024-02-05T05:26:18.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:26:18.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691878 2024-02-05T05:26:18.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:18.343 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528355 2024-02-05T05:26:18.343 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:18.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:26:18.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528355 2024-02-05T05:26:18.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:18.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528355 2024-02-05T05:26:18.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528355' 2024-02-05T05:26:18.346 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.8 seq 193273528355 2024-02-05T05:26:18.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:26:18.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528356 -lt 193273528355 2024-02-05T05:26:18.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:18.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-386547056642 2024-02-05T05:26:18.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:18.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:26:18.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-386547056642 2024-02-05T05:26:18.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:18.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-02-05T05:26:18.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 386547056642' 2024-02-05T05:26:18.618 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.9 seq 386547056642 2024-02-05T05:26:18.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:26:18.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056643 -lt 386547056642 2024-02-05T05:26:18.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:26:18.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:18.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:26:19.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:26:19.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:26:19.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:26:19.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:19.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:19.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:26:19.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:26:19.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:26:19.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:26:19.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:26:19.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269401"' 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269401' 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269403"' 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269403' 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 269401 269403' 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269404: /' 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269401 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:26:19.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269406: /' 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:26:23.176 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:26:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: remove 2#2:eb822e21:::SOMETHING:head# 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:26:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:26:23.179 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:26:23.179 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:26:23.179 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:26:23.179 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:26:23.179 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:26:23.179 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:26:23.180 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: start osd.9 2024-02-05T05:26:23.181 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:26:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:26:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:26:23.220 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:26:23.220 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:26:23.220 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:26:23.220 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:26:23.220 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:26:23.220 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:26:23.221 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:26:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:26:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:26:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:26:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:26:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:26:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:269404: start osd.5 2024-02-05T05:26:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 2024-02-05T05:26:23.207+0000 7fce076134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 2024-02-05T05:26:23.209+0000 7fce076134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 2024-02-05T05:26:23.210+0000 7fce076134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:27.870 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 0 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 2024-02-05T05:26:23.725+0000 7fce076134c0 -1 Falling back to public interface 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:27.871 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 1 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 2 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 2024-02-05T05:26:26.527+0000 7fce076134c0 -1 osd.9 92 log_to_monitors true 2024-02-05T05:26:27.872 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 3 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269406: osd.9 up in weight 1 up_from 98 up_thru 0 down_at 93 last_c269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 2024-02-05T05:26:23.246+0000 7f375981c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 2024-02-05T05:26:23.249+0000 7f375981c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 2024-02-05T05:26:23.250+0000 7f375981c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:27.873 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 0 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 2024-02-05T05:26:23.768+0000 7f375981c4c0 -1 Falling back to public interface 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.874 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 1 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 2 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 2024-02-05T05:26:26.696+0000 7f375981c4c0 -1 osd.5 92 log_to_monitors true 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:27.875 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:26:27.876 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 3 2024-02-05T05:26:27.876 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:27.876 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperlean_interval [90,92) [v2:127.0.0.1:6842/3856927885,v1:127.0.0.1:6843/3856927885] [v2:127.0.0.1:6844/3856927885,v1:127.0.0.1:6845/3856927885] exists,up 14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:26:28.830 INFO:tasks.workunit.client.0.smithi188.stderr:269406: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 1 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 2 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 3 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 4 2024-02-05T05:26:28.831 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 5 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 6 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 7 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 8 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: 9' 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:26:28.832 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-02-05T05:26:28.833 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-02-05T05:26:28.833 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043' 2024-02-05T05:26:28.833 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.387 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:26:30.387 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-02-05T05:26:30.387 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-02-05T05:26:30.387 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519' 2024-02-05T05:26:30.387 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.387 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:26:30.387 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645330 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645330 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519 3-309237645330' 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519 3-309237645330 4-107374182464' 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-02-05T05:26:30.388 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519 3-309237645330 4-107374182464 5-425201762306' 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855411 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855411 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519 3-309237645330 4-107374182464 5-425201762306 6-150323855411' 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691882 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691882 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519 3-309237645330 4-107374182464 5-425201762306 6-150323855411 7-171798691882' 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.389 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:26:30.390 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clones.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:30.390 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:30.390 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:30.390 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:26:30.390 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 4 2024-02-05T05:26:30.390 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:30.390 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: osd.5 up in weight 1 up_from 99 up_thru 99 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6874/1503433927,v1:127.0.0.1:6875/1503433927] [v2:127.0.0.1:6876/1503433927,v1:127.0.0.1:6877/1503433927] exists,up 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:26:30.391 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 1 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 2 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 3 2024-02-05T05:26:30.392 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 4 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 5 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 6 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 7 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 8 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: 9' 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-02-05T05:26:30.393 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269404: .client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528359 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528359 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519 3-309237645330 4-107374182464 5-425201762306 6-150323855411 7-171798691882 8-193273528359' 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673043 2-64424509519 3-309237645330 4-107374182464 5-425201762306 6-150323855411 7-171798691882 8-193273528359 9-420906795010' 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-02-05T05:26:31.118 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.0 seq 21474836569 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836569 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-02-05T05:26:31.119 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.1 seq 42949673043 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673043 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:31.120 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.c//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045' 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520' 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:26:31.212 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645332 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645332 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520 3-309237645332' 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182466 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182466 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520 3-309237645332 4-107374182466' 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762307 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762307 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520 3-309237645332 4-107374182466 5-425201762307' 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:26:31.213 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855412 2024-02-05T05:26:31.214 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855412 2024-02-05T05:26:31.214 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520 3-309237645332 4-107374182466 5-425201762307 6-150323855412' 2024-02-05T05:26:31.214 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:31.214 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:26:32.056 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flushlient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-02-05T05:26:32.057 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:32.057 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-02-05T05:26:32.057 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-02-05T05:26:32.057 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.2 seq 64424509519 2024-02-05T05:26:32.057 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:26:32.057 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509519 2024-02-05T05:26:32.058 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:32.058 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645330 2024-02-05T05:26:32.058 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:32.058 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:26:32.058 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645330 2024-02-05T05:26:32.058 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:32.059 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645330 2024-02-05T05:26:32.059 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645330' 2024-02-05T05:26:32.059 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.3 seq 309237645330 2024-02-05T05:26:32.059 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:26:32.059 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645330 -lt 309237645330 2024-02-05T05:26:32.059 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:32.060 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-02-05T05:26:32.060 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:32.060 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:26:32.060 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-02-05T05:26:32.060 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:32.060 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-02-05T05:26:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-02-05T05:26:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.4 seq 107374182464 2024-02-05T05:26:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:26:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182466 -lt 107374182464 2024-02-05T05:26:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-425201762306 2024-02-05T05:26:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:32.062 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:26:32.062 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-425201762306 2024-02-05T05:26:32.062 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:32.062 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-02-05T05:26:33.025 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiti_pg_stats: seq=171798691883 2024-02-05T05:26:33.025 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-02-05T05:26:33.025 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520 3-309237645332 4-107374182466 5-425201762307 6-150323855412 7-171798691883' 2024-02-05T05:26:33.025 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528360 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528360 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520 3-309237645332 4-107374182466 5-425201762307 6-150323855412 7-171798691883 8-193273528360' 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673045 2-64424509520 3-309237645332 4-107374182466 5-425201762307 6-150323855412 7-171798691883 8-193273528360 9-420906795011' 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-02-05T05:26:33.026 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.0 seq 21474836570 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836570 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836571 -lt 21474836570 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:26:33.027 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-02-05T05:26:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //homeng osd.5 seq 425201762306' 2024-02-05T05:26:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.5 seq 425201762306 2024-02-05T05:26:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:26:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762307 -lt 425201762306 2024-02-05T05:26:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855411 2024-02-05T05:26:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855411 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855411 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855411' 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.6 seq 150323855411 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855412 -lt 150323855411 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691882 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691882 2024-02-05T05:26:33.192 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691882 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.7 seq 171798691882 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691883 -lt 171798691882 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528359 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528359 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528359 2024-02-05T05:26:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528359' 2024-02-05T05:26:33.194 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.8 seq 193273528359 2024-02-05T05:26:33.194 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:26:33.194 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528360 -lt 193273528359 2024-02-05T05:26:33.941 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.1 seq 42949673045 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.2 seq 64424509520 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509520 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645332 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645332 2024-02-05T05:26:33.944 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645332 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645332' 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.3 seq 309237645332 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645332 -lt 309237645332 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182466 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182466 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182466 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182466' 2024-02-05T05:26:33.945 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.4 seq 107374182466 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clon_stats: for s in $seqs 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-420906795010 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-420906795010 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-02-05T05:26:34.631 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 420906795010' 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: waiting osd.9 seq 420906795010 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795010 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:26:34.632 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:269406: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:269406: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:26:34.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:26:35.179 INFO:tasks.workunit.client.0.smithi188.stderr:e.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:26:35.180 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182466 -lt 107374182466 2024-02-05T05:26:35.180 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:35.180 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-425201762307 2024-02-05T05:26:35.180 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:35.180 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-425201762307 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762307 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 425201762307' 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.5 seq 425201762307 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762308 -lt 425201762307 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855412 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855412 2024-02-05T05:26:35.181 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855412 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855412' 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.6 seq 150323855412 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855413 -lt 150323855412 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-02-05T05:26:35.182 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-02-05T05:26:35.183 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.7 seq 171798691883 2024-02-05T05:26:35.183 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:26:35.183 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691883 2024-02-05T05:26:35.183 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:36.348 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528360 2024-02-05T05:26:36.348 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:36.348 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:26:36.348 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528360 2024-02-05T05:26:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528360 2024-02-05T05:26:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528360' 2024-02-05T05:26:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.8 seq 193273528360 2024-02-05T05:26:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:26:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528361 -lt 193273528360 2024-02-05T05:26:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-420906795011 2024-02-05T05:26:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:26:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-420906795011 2024-02-05T05:26:36.351 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:36.351 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-02-05T05:26:36.351 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 420906795011' 2024-02-05T05:26:36.351 INFO:tasks.workunit.client.0.smithi188.stderr:269404: waiting osd.9 seq 420906795011 2024-02-05T05:26:36.351 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:26:36.351 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795011 2024-02-05T05:26:36.352 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:26:36.352 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:36.352 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:36.352 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:26:36.352 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:26:36.352 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:26:36.353 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:26:36.353 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:26:36.353 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:26:36.353 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:26:36.353 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:26:36.353 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:26:36.354 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:26:36.679 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:269404: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:269404: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269403 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:26:36.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:26:36.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:26:36.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:26:36.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:26:36.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:26:36.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:26:36.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:26:36.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:26:36.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:26:36.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:26:36.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:36.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:36.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:36.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:37.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:37.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:26:37.501 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:26:37.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:37.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:26:37.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:37.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:26:37.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:26:37.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:26:37.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:26:37.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:37.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:37.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:37.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:37.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:05.153204+0000 '>' 2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:37.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:26:38.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:26:38.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:26:38.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:26:38.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:38.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:38.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:38.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:39.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:05.153204+0000 '>' 2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:39.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:26:40.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:26:40.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:26:40.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:26:40.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:40.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:40.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:40.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:40.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:05.153204+0000 '>' 2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:40.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:26:41.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:26:41.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:26:41.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:26:41.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:41.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:41.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:41.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:41.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:05.153204+0000 '>' 2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:41.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:26:42.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:26:42.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:26:42.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:26:42.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:26:42.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:26:42.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:26:42.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:26:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:38.085916+0000 '>' 2024-02-05T05:26:05.153204+0000 2024-02-05T05:26:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:26:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:26:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:26:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:26:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:26:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272854"' 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272854' 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272856"' 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272856' 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 272854 272856' 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272854 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272857: /' 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:26:42.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272859: /' 2024-02-05T05:26:45.918 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:26:45.918 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:26:45.918 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-02-05T05:26:45.918 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:26:45.918 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:26:45.919 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: _ 2024-02-05T05:26:45.920 INFO:tasks.workunit.client.0.smithi188.stderr:272859: hinfo_key 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: snapset 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:26:45.921 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:26:45.922 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:26:45.923 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:26:45.924 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:26:45.924 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:26:45.924 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:26:45.924 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-02-05T05:26:45.924 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-02-05T05:26:45.924 INFO:tasks.workunit.client.0.smithi188.stderr:272859: start osd.9 2024-02-05T05:26:46.051 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:26:46.051 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:26:46.051 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:26:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: _ 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: hinfo_key 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: snapset 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:26:46.053 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:26:46.054 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:26:46.055 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:26:46.056 INFO:tasks.workunit.client.0.smithi188.stderr:272857: start osd.5 2024-02-05T05:26:50.475 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:26:50.475 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-02-05T05:26:50.475 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-02-05T05:26:50.475 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:26:50.475 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 2024-02-05T05:26:45.948+0000 7f508d80d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 2024-02-05T05:26:45.951+0000 7f508d80d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 2024-02-05T05:26:45.952+0000 7f508d80d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 0 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:50.476 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 2024-02-05T05:26:46.473+0000 7f508d80d4c0 -1 Falling back to public interface 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 1 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 2 2024-02-05T05:26:50.477 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 2024-02-05T05:26:49.108+0000 7f508d80d4c0 -1 osd.9 102 log_to_monitors true 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 3 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:50.478 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-02-05T05:26:51.027 INFO:tasks.workunit.client.0.smithi188.stderr:272859: osd.9 up in weight 1 up_from 107 up_thru 0 down_overy_ops 2024-02-05T05:26:51.027 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:26:51.027 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:26:51.027 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 2024-02-05T05:26:46.078+0000 7fc3206694c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 2024-02-05T05:26:46.080+0000 7fc3206694c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 2024-02-05T05:26:46.082+0000 7fc3206694c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:26:51.028 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 0 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 2024-02-05T05:26:46.600+0000 7fc3206694c0 -1 Falling back to public interface 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 1 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:51.029 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 2 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 2024-02-05T05:26:49.387+0000 7fc3206694c0 -1 osd.5 101 log_to_monitors true 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 3 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:26:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272857: osd.5 up in weight 1 up_from 107 up_thru 107 dowat 102 last_clean_interval [98,102) [v2:127.0.0.1:6842/940185786,v1:127.0.0.1:6843/940185786] [v2:127.0.0.1:6844/940185786,v1:127.0.0.1:6845/940185786] exists,up 14c5c436-f24d-4986-ac09-ac913a12d206 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:26:51.327 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 1 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 2 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 3 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 4 2024-02-05T05:26:51.328 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 5 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 6 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 7 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 8 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: 9' 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-02-05T05:26:51.329 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050' 2024-02-05T05:26:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.cn_at 102 last_clean_interval [99,101) [v2:127.0.0.1:6874/2917361897,v1:127.0.0.1:6875/2917361897] [v2:127.0.0.1:6876/2917361897,v1:127.0.0.1:6877/2917361897] exists,up 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:26:51.878 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 1 2024-02-05T05:26:51.879 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 2 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 3 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 4 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 5 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 6 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 7 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 8 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: 9' 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836577 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836577 2024-02-05T05:26:51.880 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577' 2024-02-05T05:26:51.881 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:51.881 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:26:51.881 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-02-05T05:26:51.881 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-02-05T05:26:51.881 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051' 2024-02-05T05:26:51.881 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.621 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:26:52.621 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525' 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645337 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645337 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525 3-309237645337' 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182471 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182471 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525 3-309237645337 4-107374182471' 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.622 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525 3-309237645337 4-107374182471 5-459561500674' 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855417 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855417 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525 3-309237645337 4-107374182471 5-459561500674 6-150323855417' 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691889 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691889 2024-02-05T05:26:52.623 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525 3-309237645337 4-107374182471 5-459561500674 6-150323855417 7-171798691889' 2024-02-05T05:26:52.624 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.624 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:26:52.952 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephlone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:26:52.952 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-02-05T05:26:52.952 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-02-05T05:26:52.952 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527' 2024-02-05T05:26:52.952 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645338 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645338 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-309237645338' 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-309237645338 4-107374182472' 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-02-05T05:26:52.953 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-309237645338 4-107374182472 5-459561500675' 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855418 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855418 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-309237645338 4-107374182472 5-459561500675 6-150323855418' 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691890 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691890 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-309237645338 4-107374182472 5-459561500675 6-150323855418 7-171798691890' 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:52.954 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:26:53.315 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubunttest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528366 2024-02-05T05:26:53.315 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528366 2024-02-05T05:26:53.315 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525 3-309237645337 4-107374182471 5-459561500674 6-150323855417 7-171798691889 8-193273528366' 2024-02-05T05:26:53.315 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:53.315 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673050 2-64424509525 3-309237645337 4-107374182471 5-459561500674 6-150323855417 7-171798691889 8-193273528366 9-459561500674' 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-02-05T05:26:53.316 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.0 seq 21474836576 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836576 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.1 seq 42949673050 2024-02-05T05:26:53.317 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:26:53.318 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673050 2024-02-05T05:26:53.318 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:53.318 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-02-05T05:26:53.318 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:53.318 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.2 seq 64424509525 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509525 -lt 64424509525 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645337 2024-02-05T05:26:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645337 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645337 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645337' 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.3 seq 309237645337 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:26:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645338 -lt 309237645337 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182471 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182471 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182471 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182471' 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.4 seq 107374182471 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182472 -lt 107374182471 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-459561500674 2024-02-05T05:26:54.154 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:54.154 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:26:54.154 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-459561500674 2024-02-05T05:26:54.154 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:54.154 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-02-05T05:26:54.917 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: eu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528367 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528367 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-309237645338 4-107374182472 5-459561500675 6-150323855418 7-171798691890 8-193273528367' 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836577 1-42949673051 2-64424509527 3-309237645338 4-107374182472 5-459561500675 6-150323855418 7-171798691890 8-193273528367 9-459561500675' 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836577 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:54.918 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836577 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836577 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.0 seq 21474836577 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836577 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836577 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:26:54.920 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-02-05T05:26:54.920 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:54.920 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-02-05T05:26:54.920 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-02-05T05:26:54.920 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.1 seq 42949673051 2024-02-05T05:26:54.920 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:26:55.384 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 42949cho 'waiting osd.5 seq 459561500674' 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.5 seq 459561500674 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500675 -lt 459561500674 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855417 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855417 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855417 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855417' 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.6 seq 150323855417 2024-02-05T05:26:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855417 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691889 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691889 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691889 2024-02-05T05:26:55.386 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.7 seq 171798691889 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691889 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528366 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528366 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528366 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528366' 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.8 seq 193273528366 2024-02-05T05:26:55.387 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:26:55.388 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528367 -lt 193273528366 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210673051 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.825 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.2 seq 64424509527 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509527 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-309237645338 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-309237645338 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645338 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 309237645338' 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.3 seq 309237645338 2024-02-05T05:26:55.826 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645338 -lt 309237645338 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.4 seq 107374182472 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182473 -lt 107374182472 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-459561500675 2024-02-05T05:26:55.827 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-hel: flush_pg_stats: for s in $seqs 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500674 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500674 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500674' 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: waiting osd.9 seq 459561500674 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500675 -lt 459561500674 2024-02-05T05:26:56.657 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:56.658 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:26:56.659 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:26:56.659 INFO:tasks.workunit.client.0.smithi188.stderr:272859: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:26:56.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:pers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-459561500675 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 459561500675' 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.5 seq 459561500675 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500675 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855418 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:56.684 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855418 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855418 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855418' 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.6 seq 150323855418 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855419 -lt 150323855418 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691890 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691890 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691890 2024-02-05T05:26:56.685 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.7 seq 171798691890 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691890 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528367 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528367 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528367 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.8 seq 193273528367 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528367 -lt 193273528367 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500675 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500675 2024-02-05T05:26:58.306 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500675' 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: waiting osd.9 seq 459561500675 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500675 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:26:58.307 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:272857: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:26:58.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272856 2024-02-05T05:26:58.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:26:58.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:26:58.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:26:58.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:26:58.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:26:58.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:26:58.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:26:58.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-02-05T05:26:58.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-02-05T05:26:58.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 276110"' 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 276110' 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 276112"' 2024-02-05T05:26:58.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 276112' 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 276110 276112' 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 276110 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/276113: /' 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:26:58.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/276115: /' 2024-02-05T05:27:01.383 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.383 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:01.383 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:27:01.383 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:01.383 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:27:01.383 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-02-05T05:27:01.384 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: remove 1#2:eb822e21:::SOMETHING:head# 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:27:01.385 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:01.386 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:01.386 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:01.386 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:01.386 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:27:01.386 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:01.387 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: start osd.5 2024-02-05T05:27:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:01.460 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.460 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:01.460 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:27:01.460 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T05:27:01.461 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: remove 0#2:eb822e21:::SOMETHING:head# 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:01.462 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:01.463 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:01.463 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:01.463 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:27:01.463 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:27:01.463 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:01.463 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:01.463 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:01.464 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:01.465 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:01.465 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:01.465 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:01.465 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:27:01.465 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:27:01.465 INFO:tasks.workunit.client.0.smithi188.stderr:276113: start osd.3 2024-02-05T05:27:01.465 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:05.890 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 2024-02-05T05:27:01.414+0000 7f577d5374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 2024-02-05T05:27:01.416+0000 7f577d5374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 2024-02-05T05:27:01.418+0000 7f577d5374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:05.891 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 0 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 2024-02-05T05:27:01.945+0000 7f577d5374c0 -1 Falling back to public interface 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 1 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:05.892 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 2 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 2024-02-05T05:27:04.808+0000 7f577d5374c0 -1 osd.5 109 log_to_monitors true 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 3 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:05.893 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:06.134 INFO:tasks.workunit.client.0.smithi188.stderr:276115: osd.5 up in weight 1 up_from 113 up_thru 113 down_at 110 l276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 2024-02-05T05:27:01.490+0000 7f459a00d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 2024-02-05T05:27:01.493+0000 7f459a00d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 2024-02-05T05:27:01.494+0000 7f459a00d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:06.135 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 0 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 2024-02-05T05:27:02.009+0000 7f459a00d4c0 -1 Falling back to public interface 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 1 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:06.136 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 2 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 2024-02-05T05:27:04.805+0000 7f459a00d4c0 -1 osd.3 109 log_to_monitors true 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 3 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:06.137 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:06.658 INFO:tasks.workunit.client.0.smithi188.stderr:276113: osd.3 up in weight 1 up_from 113 up_thru 113 down_at 110 last_clean_interval [107,109) [v2:127.0.0.1:6826/1686819896,v1:127.0.0.1:6827/1686819896] [v2:127.0.0.1:6828/1686819896,v1:127.0.0.1:6829/1686819896] exists,up 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:27:06.659 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 1 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 2 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 3 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 4 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 5 2024-02-05T05:27:06.660 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 6 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 7 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 8 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: 9' 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-02-05T05:27:06.661 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055' 2024-02-05T05:27:06.662 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:06.767 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.clieast_clean_interval [72,109) [v2:127.0.0.1:6874/2233049965,v1:127.0.0.1:6875/2233049965] [v2:127.0.0.1:6876/2233049965,v1:127.0.0.1:6877/2233049965] exists,up f68a09e8-d5b4-4da0-8457-46288076ddeb 2024-02-05T05:27:06.767 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:27:06.767 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:27:06.767 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:27:06.767 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:27:06.767 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:27:06.767 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:27:06.768 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 1 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 2 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 3 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 4 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 5 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 6 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 7 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 8 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: 9' 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836582 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836582 2024-02-05T05:27:06.769 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582' 2024-02-05T05:27:06.770 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:06.770 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:06.770 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-02-05T05:27:06.770 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-02-05T05:27:06.770 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056' 2024-02-05T05:27:06.770 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.cliennt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530' 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-02-05T05:27:07.412 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530 3-485331304450' 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530 3-485331304450 4-107374182476' 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530 3-485331304450 4-107374182476 5-485331304450' 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.413 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855422 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855422 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530 3-485331304450 4-107374182476 5-485331304450 6-150323855422' 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530 3-485331304450 4-107374182476 5-485331304450 6-150323855422 7-171798691894' 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.414 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:27:07.519 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532' 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-485331304451' 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182477 2024-02-05T05:27:07.520 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182477 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-485331304451 4-107374182477' 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-485331304451 4-107374182477 5-485331304451' 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855423 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855423 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-485331304451 4-107374182477 5-485331304451 6-150323855423' 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691895 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691895 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-485331304451 4-107374182477 5-485331304451 6-150323855423 7-171798691895' 2024-02-05T05:27:07.521 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:07.522 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528372 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528372 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-485331304451 4-107374182477 5-485331304451 6-150323855423 7-171798691895 8-193273528372' 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500680 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500680 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836582 1-42949673056 2-64424509532 3-485331304451 4-107374182477 5-485331304451 6-150323855423 7-171798691895 8-193273528372 9-459561500680' 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836582 2024-02-05T05:27:08.229 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836582 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836582 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.0 seq 21474836582 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836582 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-02-05T05:27:08.230 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.1 seq 42949673056 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673056 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:08.231 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:09.055 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.2 seq 64424509532 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-485331304451 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-485331304451 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 485331304451' 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.3 seq 485331304451 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304451 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182477 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182477 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182477 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182477' 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.4 seq 107374182477 2024-02-05T05:27:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182477 -lt 107374182477 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304451 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304451 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-02-05T05:27:09.407 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo t/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528371 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528371 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530 3-485331304450 4-107374182476 5-485331304450 6-150323855422 7-171798691894 8-193273528371' 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500679 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500679 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673055 2-64424509530 3-485331304450 4-107374182476 5-485331304450 6-150323855422 7-171798691894 8-193273528371 9-459561500679' 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-02-05T05:27:09.408 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.0 seq 21474836581 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836581 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836582 -lt 21474836581 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:09.409 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-02-05T05:27:09.410 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:09.410 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-02-05T05:27:09.410 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-02-05T05:27:09.410 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.1 seq 42949673055 2024-02-05T05:27:09.410 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:09.410 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673055 2024-02-05T05:27:10.272 INFO:tasks.workunit.client.0.smithi188.stderr:27'waiting osd.5 seq 485331304451' 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.5 seq 485331304451 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304451 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855423 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855423 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855423 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855423' 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.6 seq 150323855423 2024-02-05T05:27:10.273 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855423 -lt 150323855423 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691895 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691895 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691895 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.7 seq 171798691895 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691895 2024-02-05T05:27:10.274 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528372 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528372 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528372 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.8 seq 193273528372 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:27:10.275 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528372 -lt 193273528372 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fl6115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-02-05T05:27:10.374 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.2 seq 64424509530 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509530 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-485331304450 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-485331304450 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 485331304450' 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.3 seq 485331304450 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304450 2024-02-05T05:27:10.375 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.4 seq 107374182476 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182477 -lt 107374182476 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304450 2024-02-05T05:27:10.377 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:11.300 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:27:11.300 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304450 2024-02-05T05:27:11.300 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 485331304450' 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.5 seq 485331304450 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304450 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855422 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855422 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855422 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855422' 2024-02-05T05:27:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.6 seq 150323855422 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855424 -lt 150323855422 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.7 seq 171798691894 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691894 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:11.302 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528371 2024-02-05T05:27:11.303 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:11.303 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:27:11.303 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528371 2024-02-05T05:27:11.303 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:11.303 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528371 2024-02-05T05:27:11.683 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.cush_pg_stats: for s in $seqs 2024-02-05T05:27:11.683 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500680 2024-02-05T05:27:11.683 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:11.683 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:27:11.683 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500680 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500680 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500680' 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: waiting osd.9 seq 459561500680 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500680 -lt 459561500680 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:27:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:27:11.686 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:11.686 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:12.347 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:276113: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:276113: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:27:12.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 276112 2024-02-05T05:27:12.870 INFO:tasks.workunit.client.0.smithi188.stderr:lient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528371' 2024-02-05T05:27:12.870 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.8 seq 193273528371 2024-02-05T05:27:12.870 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:27:12.870 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528372 -lt 193273528371 2024-02-05T05:27:12.870 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500679 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500679 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500679 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500679' 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: waiting osd.9 seq 459561500679 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500681 -lt 459561500679 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:12.871 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:276115: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T05:27:12.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:27:13.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-02-05T05:27:13.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-02-05T05:27:13.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:27:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:27:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:13.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:13.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:26:38.085916+0000 2024-02-05T05:27:13.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:27:13.664 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-02-05T05:27:13.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:26:38.085916+0000 2024-02-05T05:27:13.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:27:13.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:26:38.085916+0000 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:13.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:14.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:38.085916+0000 '>' 2024-02-05T05:26:38.085916+0000 2024-02-05T05:27:14.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:15.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:15.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:15.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:15.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:15.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:15.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:15.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:15.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:26:38.085916+0000 '>' 2024-02-05T05:26:38.085916+0000 2024-02-05T05:27:15.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:27:16.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:27:16.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:27:16.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:27:16.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:27:16.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:27:16.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:27:16.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:27:16.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:27:13.770829+0000 '>' 2024-02-05T05:26:38.085916+0000 2024-02-05T05:27:16.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:27:16.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-02-05T05:27:16.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:16.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:27:16.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 279538"' 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 279538' 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 279540"' 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 279540' 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 279538 279540' 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 279538 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/279541: /' 2024-02-05T05:27:16.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:27:16.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/279543: /' 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T05:27:19.396 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: _ 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: hinfo_key 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: snapset 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T05:27:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:19.399 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:19.399 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:19.399 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:19.399 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T05:27:19.399 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T05:27:19.399 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:19.399 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:19.400 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:19.401 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:19.401 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:19.401 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:19.401 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:19.401 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T05:27:19.401 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T05:27:19.401 INFO:tasks.workunit.client.0.smithi188.stderr:279541: start osd.3 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-02-05T05:27:19.784 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: _ 2024-02-05T05:27:19.785 INFO:tasks.workunit.client.0.smithi188.stderr:279543: hinfo_key 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: snapset 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-02-05T05:27:19.786 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:19.787 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:19.788 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:19.788 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:19.788 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:27:19.788 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:27:19.788 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-02-05T05:27:19.788 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-02-05T05:27:19.788 INFO:tasks.workunit.client.0.smithi188.stderr:279543: start osd.5 2024-02-05T05:27:23.567 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 2024-02-05T05:27:19.426+0000 7f456200e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 2024-02-05T05:27:19.428+0000 7f456200e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 2024-02-05T05:27:19.430+0000 7f456200e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:23.568 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 0 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 2024-02-05T05:27:19.946+0000 7f456200e4c0 -1 Falling back to public interface 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 1 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:23.569 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 2 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 2024-02-05T05:27:22.366+0000 7f456200e4c0 -1 osd.3 114 log_to_monitors true 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 3 2024-02-05T05:27:23.570 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:24.242 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 2024-02-05T05:27:19.808+0000 7f1d95fe24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 2024-02-05T05:27:19.815+0000 7f1d95fe24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 2024-02-05T05:27:19.816+0000 7f1d95fe24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 0 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 2024-02-05T05:27:20.331+0000 7f1d95fe24c0 -1 Falling back to public interface 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 1 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:24.244 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 2 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 2024-02-05T05:27:23.272+0000 7f1d95fe24c0 -1 osd.5 114 log_to_monitors true 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 3 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:24.245 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-02-05T05:27:24.968 INFO:tasks.workunit.client.0.smithi188.stderr:279543: osd.5 up in weight 1 up_from 120 up_thru 113 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T05:27:24.968 INFO:tasks.workunit.client.0.smithi188.stderr:279541: osd.3 up in weight 1 up_from 118 up_thru 113 down_at 115 last_clean_interval [113,114) [v2:127.0.0.1:6826/765900579,v1:127.0.0.1:6827/765900579] [v2:127.0.0.1:6828/765900579,v1:127.0.0.1:6829/765900579] exists,up f68a09e8-d5b4-4da0-8457-46288076ddeb 2024-02-05T05:27:24.968 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:27:24.968 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:27:24.968 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:27:24.968 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:27:24.968 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:27:24.969 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 1 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 2 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 3 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 4 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 5 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 6 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 7 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 8 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: 9' 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836586 2024-02-05T05:27:24.970 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836586 2024-02-05T05:27:24.971 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586' 2024-02-05T05:27:24.971 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:24.971 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:24.971 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673061 2024-02-05T05:27:24.971 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673061 2024-02-05T05:27:24.971 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061' 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/n_at 115 last_clean_interval [113,114) [v2:127.0.0.1:6874/3890834153,v1:127.0.0.1:6875/3890834153] [v2:127.0.0.1:6876/3890834153,v1:127.0.0.1:6877/3890834153] exists,up 7c8d6b1d-ddd9-4653-83d9-11cedcdd719c 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:27:25.212 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 1 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 2 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 3 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 4 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 5 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 6 2024-02-05T05:27:25.213 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 7 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 8 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: 9' 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836587 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836587 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587' 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673062 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673062 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062' 2024-02-05T05:27:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.973 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536' 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140930 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140930 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536 3-506806140930' 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:27:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182482 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182482 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536 3-506806140930 4-107374182482' 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075522 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075522 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536 3-506806140930 4-107374182482 5-515396075522' 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855428 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855428 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536 3-506806140930 4-107374182482 5-515396075522 6-150323855428' 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:27:25.975 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691899 2024-02-05T05:27:25.976 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-02-05T05:27:25.976 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536 3-506806140930 4-107374182482 5-515396075522 6-150323855428 7-171798691899' 2024-02-05T05:27:25.976 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.204 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/stclone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:27:26.204 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-02-05T05:27:26.204 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-02-05T05:27:26.204 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537' 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=506806140931 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 506806140931 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537 3-506806140931' 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182483 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182483 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537 3-506806140931 4-107374182483' 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=515396075523 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 515396075523 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537 3-506806140931 4-107374182483 5-515396075523' 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855429 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855429 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537 3-506806140931 4-107374182483 5-515396075523 6-150323855429' 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691900 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537 3-506806140931 4-107374182483 5-515396075523 6-150323855429 7-171798691900' 2024-02-05T05:27:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.207 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:27:26.796 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubunandalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-02-05T05:27:26.796 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528376 2024-02-05T05:27:26.796 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528376 2024-02-05T05:27:26.796 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536 3-506806140930 4-107374182482 5-515396075522 6-150323855428 7-171798691899 8-193273528376' 2024-02-05T05:27:26.796 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500685 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500685 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836586 1-42949673061 2-64424509536 3-506806140930 4-107374182482 5-515396075522 6-150323855428 7-171798691899 8-193273528376 9-459561500685' 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836586 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836586 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836586 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.0 seq 21474836586 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:26.797 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836588 -lt 21474836586 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673061 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673061 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673061 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.1 seq 42949673061 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673062 -lt 42949673061 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-02-05T05:27:26.798 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtestu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528377 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528377 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537 3-506806140931 4-107374182483 5-515396075523 6-150323855429 7-171798691900 8-193273528377' 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500686 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500686 2024-02-05T05:27:26.940 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836587 1-42949673062 2-64424509537 3-506806140931 4-107374182483 5-515396075523 6-150323855429 7-171798691900 8-193273528377 9-459561500686' 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836587 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836587 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836587 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.0 seq 21474836587 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836588 -lt 21474836587 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673062 2024-02-05T05:27:26.941 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673062 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673062 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.1 seq 42949673062 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673062 -lt 42949673062 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:26.942 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:27.818 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntut/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:27:27.818 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.2 seq 64424509536 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509536 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140930 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140930 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140930 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140930' 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.3 seq 506806140930 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140930 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182482 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182482 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182482 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182482' 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.4 seq 107374182482 2024-02-05T05:27:27.820 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:27:27.821 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182483 -lt 107374182482 2024-02-05T05:27:27.821 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:27.821 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075522 2024-02-05T05:27:27.821 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:27.821 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:27:27.821 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075522 2024-02-05T05:27:27.821 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-02-05T05:27:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-02-05T05:27:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-02-05T05:27:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.2 seq 64424509537 2024-02-05T05:27:27.885 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-506806140931 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-506806140931 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=506806140931 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 506806140931' 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.3 seq 506806140931 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 506806140931 -lt 506806140931 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:27.886 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182483 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182483 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182483 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.4 seq 107374182483 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182483 -lt 107374182483 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-515396075523 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-515396075523 2024-02-05T05:27:27.887 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:27.888 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=515396075523 2024-02-05T05:27:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075523' 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.5 seq 515396075523 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075523 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855429 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855429 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855429 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855429' 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.6 seq 150323855429 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:27:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855429 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.7 seq 171798691900 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691900 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528377 2024-02-05T05:27:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528377 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528377 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528377' 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.8 seq 193273528377 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:27:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528378 -lt 193273528377 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.5522 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 515396075522' 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.5 seq 515396075522 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 515396075523 -lt 515396075522 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855428 2024-02-05T05:27:29.111 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855428 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855428 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855428' 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.6 seq 150323855428 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855429 -lt 150323855428 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-02-05T05:27:29.112 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.7 seq 171798691899 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691901 -lt 171798691899 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528376 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528376 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528376 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528376' 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.8 seq 193273528376 2024-02-05T05:27:29.113 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-02-05T05:27:30.323 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 19327sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:30.323 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500686 2024-02-05T05:27:30.323 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:30.323 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:27:30.323 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500686 2024-02-05T05:27:30.323 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:30.323 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500686 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500686' 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: waiting osd.9 seq 459561500686 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500686 -lt 459561500686 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:30.324 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:279543: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:27:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:3528378 -lt 193273528376 2024-02-05T05:27:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:27:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500685 2024-02-05T05:27:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500685 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500685 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500685' 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: waiting osd.9 seq 459561500685 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500686 -lt 459561500685 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:27:30.426 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:279541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T05:27:30.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T05:27:30.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 279540 2024-02-05T05:27:30.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T05:27:30.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T05:27:30.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T05:27:30.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-02-05T05:27:30.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-02-05T05:27:30.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:27:30.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:30.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:30.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:27:30.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:27:30.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:27:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:27:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:27:30.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:27:30.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:27:30.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:27:30.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:27:30.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:27:30.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:27:30.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:27:30.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:27:30.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:27:30.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:30.684 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:30.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:27:30.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:27:30.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:27:30.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:27:30.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:27:30.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:27:30.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:27:30.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:27:30.688 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:27:30.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:27:30.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:27:30.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:27:30.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:27:30.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:27:30.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:27:30.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:27:30.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:27:30.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:27:30.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:27:30.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:27:30.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:27:30.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:27:30.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:30.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:30.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:27:30.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:27:30.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:27:30.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:27:30.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:27:30.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:30.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:30.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:27:30.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:27:30.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:27:30.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:27:30.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:27:30.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T05:27:31.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:27:31.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:31.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:31.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:31.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:31.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:31.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:31.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T05:27:31.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:27:31.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:27:31.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:27:31.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:27:31.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:27:31.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:27:31.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:27:31.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-02-05T05:27:31.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:27:31.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:27:31.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:27:31.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:27:31.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:27:31.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:27:32.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:27:32.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:32.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:32.234 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:32.235 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:32.235 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:32.235 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:32.235 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:27:32.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:27:32.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:27:32.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:27:32.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4317f3a0-f0d4-4101-b98c-4273a8b48cff 2024-02-05T05:27:32.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4317f3a0-f0d4-4101-b98c-4273a8b48cff' 2024-02-05T05:27:32.270 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 4317f3a0-f0d4-4101-b98c-4273a8b48cff 2024-02-05T05:27:32.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:27:32.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEccBlihDcEBAAZrbup9T1Uq/O4HVnjkffUA== 2024-02-05T05:27:32.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEccBlihDcEBAAZrbup9T1Uq/O4HVnjkffUA=="}' 2024-02-05T05:27:32.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4317f3a0-f0d4-4101-b98c-4273a8b48cff -i td/osd-scrub-repair/0/new.json 2024-02-05T05:27:32.490 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:27:32.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:27:32.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEccBlihDcEBAAZrbup9T1Uq/O4HVnjkffUA== --osd-uuid 4317f3a0-f0d4-4101-b98c-4273a8b48cff 2024-02-05T05:27:32.529 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:32.528+0000 7fde3f68c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:32.540 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:32.539+0000 7fde3f68c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:32.543 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:32.540+0000 7fde3f68c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:32.543 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:32.541+0000 7fde3f68c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:27:32.543 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:32.541+0000 7fde3f68c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:27:32.543 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:32.541+0000 7fde3f68c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:27:32.633 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:32.631+0000 7fde3f68c4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:27:38.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:27:38.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:27:38.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:27:38.330 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:27:38.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:27:38.678 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:27:38.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:27:38.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:27:38.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:27:38.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:27:38.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:27:38.704 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:38.702+0000 7f9d28f4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:38.705 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:38.704+0000 7f9d28f4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:38.707 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:38.706+0000 7f9d28f4a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:27:38.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:27:38.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:27:38.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:27:38.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:27:38.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:27:38.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:38.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:27:38.948 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:27:38.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:38.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:39.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:39.227 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:39.224+0000 7f9d28f4a4c0 -1 Falling back to public interface 2024-02-05T05:27:40.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:40.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:40.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:27:40.218 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:27:40.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:40.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:40.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:41.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:41.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:41.496 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:27:41.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:27:41.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:41.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:41.599 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:27:41.598+0000 7f9d28f4a4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:27:41.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:42.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:42.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:42.775 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:27:42.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:27:42.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:42.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:43.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:44.054 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:27:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:27:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:44.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:45.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:45.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:45.345 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:27:45.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:27:45.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:45.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:45.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:46.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:46.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:46.625 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:27:46.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:27:46.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:46.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:46.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:47.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:47.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:47.904 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:27:47.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:27:47.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:47.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:48.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:49.186 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:27:49.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:27:49.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:49.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:49.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:50.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:50.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:50.531 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:27:50.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:27:50.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:50.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:50.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:51.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:51.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:51.812 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:27:51.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:27:51.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:51.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:52.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:53.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:53.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:53.180 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:27:53.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:27:53.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:53.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:53.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:54.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:54.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:54.456 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:27:54.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:27:54.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:54.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:54.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:55.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:55.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:55.889 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:27:55.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:27:55.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:55.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:56.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:57.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:57.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:57.209 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:27:57.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:27:57.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:57.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:57.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:27:58.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:27:58.861 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:27:58.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:27:58.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:27:58.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:27:58.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:27:59.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:00.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:00.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:00.489 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:28:00.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:28:00.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:00.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:28:01.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:02.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:02.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:02.282 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:28:02.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:28:02.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:02.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:28:02.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:03.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:03.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:03.754 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:28:03.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:28:03.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:03.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:28:04.091 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/849634167,v1:127.0.0.1:6803/849634167] [v2:127.0.0.1:6804/849634167,v1:127.0.0.1:6805/849634167] exists,up 4317f3a0-f0d4-4101-b98c-4273a8b48cff 2024-02-05T05:28:04.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:04.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:04.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:04.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-02-05T05:28:04.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:04.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:28:04.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:28:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86933c9f-8c29-4504-bca3-83bbdb48f121 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 86933c9f-8c29-4504-bca3-83bbdb48f121' 2024-02-05T05:28:04.098 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 86933c9f-8c29-4504-bca3-83bbdb48f121 2024-02-05T05:28:04.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:28:04.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkccBlKzeuBhAA2zoyzhFNJRzCrtLwhVlitQ== 2024-02-05T05:28:04.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkccBlKzeuBhAA2zoyzhFNJRzCrtLwhVlitQ=="}' 2024-02-05T05:28:04.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86933c9f-8c29-4504-bca3-83bbdb48f121 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:28:04.479 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:28:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:28:04.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkccBlKzeuBhAA2zoyzhFNJRzCrtLwhVlitQ== --osd-uuid 86933c9f-8c29-4504-bca3-83bbdb48f121 2024-02-05T05:28:04.520 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:04.519+0000 7f6955a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:04.556 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:04.554+0000 7f6955a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:04.557 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:04.555+0000 7f6955a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:04.557 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:04.556+0000 7f6955a144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:28:04.557 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:04.556+0000 7f6955a144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:28:04.557 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:04.556+0000 7f6955a144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:28:04.632 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:04.631+0000 7f6955a144c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:28:10.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:28:10.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:28:10.589 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:28:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:28:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:28:10.937 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:28:10.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:28:10.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:28:10.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:28:10.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:28:10.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:28:10.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:10.962+0000 7fe46c0674c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:10.970 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:10.964+0000 7fe46c0674c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:10.971 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:10.965+0000 7fe46c0674c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:11.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:28:11.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:11.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:28:11.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:11.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:11.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:11.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:11.205 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:28:11.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:11.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:11.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:11.492 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:11.491+0000 7fe46c0674c0 -1 Falling back to public interface 2024-02-05T05:28:12.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:12.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:12.474 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:28:12.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:12.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:12.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:12.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:13.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:13.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:13.753 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:28:13.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:13.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:13.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:13.872 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:13.871+0000 7fe46c0674c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:28:14.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:15.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:15.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:15.074 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:28:15.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:15.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:15.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:15.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:16.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:16.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:16.354 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:28:16.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:28:16.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:16.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:16.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:17.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:17.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:17.643 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:28:17.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:28:17.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:17.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:17.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:18.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:18.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:18.924 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:28:18.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:28:18.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:18.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:19.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:20.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:20.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:20.202 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:28:20.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:28:20.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:20.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:20.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:21.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:21.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:21.601 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:28:21.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:28:21.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:21.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:21.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:22.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:22.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:22.894 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:28:22.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:28:22.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:22.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:23.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:24.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:24.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:24.175 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:28:24.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:28:24.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:24.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:24.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:25.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:25.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:25.577 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:28:25.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:28:25.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:25.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:25.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:26.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:26.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:26.858 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:28:26.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:28:26.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:26.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:27.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:28.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:28.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:28.508 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:28:28.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:28:28.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:28.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:28.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:29.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:29.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:29.901 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:28:29.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:28:29.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:29.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:30.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:31.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:31.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:31.297 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:28:31.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:28:31.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:31.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:31.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:32.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:32.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:32.990 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:28:32.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:28:32.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:32.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:33.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:34.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:34.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:34.653 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:28:34.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:28:34.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:34.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:35.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:36.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:36.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:36.019 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:28:36.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:28:36.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:36.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:36.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:37.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:37.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:37.463 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:28:37.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:28:37.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:37.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:28:37.732 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3538884769,v1:127.0.0.1:6811/3538884769] [v2:127.0.0.1:6812/3538884769,v1:127.0.0.1:6813/3538884769] exists,up 86933c9f-8c29-4504-bca3-83bbdb48f121 2024-02-05T05:28:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-02-05T05:28:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:28:38.000 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:28:38.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:28:38.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:28:38.366 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:28:38.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:28:39.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:28:41.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-02-05T05:28:41.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:41.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:41.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:41.237 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:41.237 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:41.237 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:41.237 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:41.237 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:41.237 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:41.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:41.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:41.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:41.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:41.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:41.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:41.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:41.597 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:28:41.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:41.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:41.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:41.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-02-05T05:28:41.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-02-05T05:28:41.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-02-05T05:28:41.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:41.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:41.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-02-05T05:28:41.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-02-05T05:28:41.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2024-02-05T05:28:41.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:41.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-02-05T05:28:41.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:41.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:41.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-02-05T05:28:41.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:41.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-02-05T05:28:41.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-02-05T05:28:41.805 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836491 2024-02-05T05:28:41.805 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:42.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-02-05T05:28:42.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:43.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:43.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:43.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-02-05T05:28:43.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:43.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-02-05T05:28:43.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:43.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:43.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-02-05T05:28:43.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:43.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-02-05T05:28:43.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-02-05T05:28:43.350 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672970 2024-02-05T05:28:43.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:43.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-02-05T05:28:43.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:43.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:43.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:43.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:44.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:28:44.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:44.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:44.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:28:44.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:28:44.932 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:28:44.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:28:45.451 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:28:45.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:28:45.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:28:45.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:28:45.715 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:28:45.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:28:45.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:28:45.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:28:46.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:28:46.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:28:46.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T05:28:47.856 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:28:48.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:28:48.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:28:48.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:28:48.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:28:48.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:28:48.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:28:48.374 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:28:48.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:28:48.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:28:48.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:28:48.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:28:48.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:28:48.376 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:28:48.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:28:48.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:28:48.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:28:48.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:28:48.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:28:48.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:28:48.402 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:48.401+0000 7f32c22164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:48.415 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:48.413+0000 7f32c22164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:48.416 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:48.414+0000 7f32c22164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:28:48.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:28:48.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:48.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:28:48.930 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:48.929+0000 7f32c22164c0 -1 Falling back to public interface 2024-02-05T05:28:48.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:49.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:49.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:49.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:28:49.941 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:28:49.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:49.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:28:50.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:51.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:51.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:51.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:28:51.213 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:28:51.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:51.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:28:51.263 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:28:51.262+0000 7f32c22164c0 -1 osd.0 22 log_to_monitors true 2024-02-05T05:28:51.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:28:52.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:28:52.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:28:52.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:28:52.499 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:28:52.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:28:52.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:28:52.860 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/420270478,v1:127.0.0.1:6803/420270478] [v2:127.0.0.1:6804/420270478,v1:127.0.0.1:6805/420270478] exists,up 4317f3a0-f0d4-4101-b98c-4273a8b48cff 2024-02-05T05:28:52.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:28:52.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:28:52.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:28:52.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:28:52.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:28:52.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:28:52.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:28:52.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:28:52.861 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:28:52.861 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:28:52.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:28:52.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:28:52.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:28:52.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:28:52.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:28:52.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:28:52.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:28:52.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:28:52.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:28:53.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:28:53.233 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:28:53.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:28:53.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:53.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:28:53.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-02-05T05:28:53.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-02-05T05:28:53.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-02-05T05:28:53.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:28:53.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:28:53.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-05T05:28:53.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-05T05:28:53.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672974' 2024-02-05T05:28:53.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:53.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-02-05T05:28:53.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:53.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:28:53.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-02-05T05:28:53.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:53.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-02-05T05:28:53.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-02-05T05:28:53.441 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 111669149698 2024-02-05T05:28:53.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:53.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-02-05T05:28:53.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:28:54.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:28:54.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:28:54.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-02-05T05:28:54.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:28:54.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-05T05:28:54.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:28:54.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:28:54.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-05T05:28:54.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:28:54.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-05T05:28:54.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-05T05:28:54.988 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672974 2024-02-05T05:28:54.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:28:55.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-02-05T05:28:55.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:28:55.254 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:55.254 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:55.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:28:55.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:28:55.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:28:55.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:28:55.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:28:55.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:28:55.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:28:55.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:28:55.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:28:55.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:28:55.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:28:55.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:28:56.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:28:56.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:28:56.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:28:56.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-02-05T05:28:56.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T05:28:56.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:28:56.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T05:28:56.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:28:56.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-02-05T05:28:56.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-02-05T05:28:56.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-02-05T05:28:56.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-02-05T05:28:56.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:28:56.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:56.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:56.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:28:56.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:28:38.315378+0000 2024-02-05T05:28:56.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-02-05T05:28:57.015 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.3 on osd.1 to repair 2024-02-05T05:28:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-02-05T05:28:38.315378+0000 2024-02-05T05:28:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T05:28:57.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:28:38.315378+0000 2024-02-05T05:28:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:28:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:28:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:57.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:28:57.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:28:57.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:57.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:57.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:28:57.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:38.315378+0000 '>' 2024-02-05T05:28:38.315378+0000 2024-02-05T05:28:57.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:28:58.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:28:58.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:58.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:28:58.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:28:58.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:58.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:58.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:28:58.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:38.315378+0000 '>' 2024-02-05T05:28:38.315378+0000 2024-02-05T05:28:58.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:28:59.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:28:59.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:28:59.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:28:59.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:28:59.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:28:59.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:28:59.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:28:59.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:38.315378+0000 '>' 2024-02-05T05:28:38.315378+0000 2024-02-05T05:28:59.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:00.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:00.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:00.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:00.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:00.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:00.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:00.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:01.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:38.315378+0000 '>' 2024-02-05T05:28:38.315378+0000 2024-02-05T05:29:01.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:02.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:02.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:02.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:02.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:02.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:02.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:02.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:02.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:38.315378+0000 '>' 2024-02-05T05:28:38.315378+0000 2024-02-05T05:29:02.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:03.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:03.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:03.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:03.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:03.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:03.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:03.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:03.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:38.315378+0000 '>' 2024-02-05T05:28:38.315378+0000 2024-02-05T05:29:03.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:04.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:04.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:04.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:04.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:04.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:04.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:04.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:04.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:57.940253+0000 '>' 2024-02-05T05:28:38.315378+0000 2024-02-05T05:29:04.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:04.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:04.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:04.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:05.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:05.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:05.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:29:05.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:29:05.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:29:05.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:29:05.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:05.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:29:06.531 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:29:06.531 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:29:06.532 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:06.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:06.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:29:06.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:29:06.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:29:06.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:29:06.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:29:06.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:29:06.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:29:06.830 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:06.828+0000 7f8e3d3174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:06.832 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:06.831+0000 7f8e3d3174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:06.833 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:06.832+0000 7f8e3d3174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:07.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:29:07.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:29:07.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:29:07.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:29:07.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:29:07.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:07.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:29:07.078 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:29:07.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:07.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:07.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:07.352 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:07.351+0000 7f8e3d3174c0 -1 Falling back to public interface 2024-02-05T05:29:08.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:08.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:08.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:29:08.346 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:29:08.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:08.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:08.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:09.530 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:09.529+0000 7f8e3d3174c0 -1 osd.1 28 log_to_monitors true 2024-02-05T05:29:09.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:09.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:09.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:29:09.618 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:29:09.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:09.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:09.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:29:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:29:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:11.184 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/421619520,v1:127.0.0.1:6811/421619520] [v2:127.0.0.1:6812/421619520,v1:127.0.0.1:6813/421619520] exists,up 86933c9f-8c29-4504-bca3-83bbdb48f121 2024-02-05T05:29:11.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:29:11.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:29:11.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:29:11.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:29:11.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:29:11.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:29:11.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:29:11.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:29:11.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:29:11.536 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:29:11.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:29:11.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:11.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:29:11.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149703 2024-02-05T05:29:11.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149703 2024-02-05T05:29:11.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703' 2024-02-05T05:29:11.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:11.640 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:29:11.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-02-05T05:29:11.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-02-05T05:29:11.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149703 1-137438953474' 2024-02-05T05:29:11.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:11.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149703 2024-02-05T05:29:11.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:11.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:29:11.746 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149703 2024-02-05T05:29:11.746 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:11.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149703 2024-02-05T05:29:11.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149703' 2024-02-05T05:29:11.748 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 111669149703 2024-02-05T05:29:11.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:12.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149701 -lt 111669149703 2024-02-05T05:29:12.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:29:13.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:29:13.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:13.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149703 -lt 111669149703 2024-02-05T05:29:13.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:13.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-02-05T05:29:13.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:13.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:29:13.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-02-05T05:29:13.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:13.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-02-05T05:29:13.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-02-05T05:29:13.296 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 137438953474 2024-02-05T05:29:13.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:29:13.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-02-05T05:29:13.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:29:13.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:13.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:29:13.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:29:14.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:29:14.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:29:14.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:14.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:14.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:29:14.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:29:14.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:29:14.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:29:14.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:29:14.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-02-05T05:29:14.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T05:29:14.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T05:29:14.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T05:29:14.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:29:14.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-02-05T05:29:14.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-02-05T05:29:14.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-02-05T05:29:14.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-02-05T05:29:14.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:14.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:15.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:15.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-02-05T05:29:15.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:29:15.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:29:15.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:29:15.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:29:15.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:15.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-02-05T05:29:16.419 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:29:16.939 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:29:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:29:16.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:29:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:29:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:29:16.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:29:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:29:16.966 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:16.965+0000 7f09537fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:16.968 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:16.967+0000 7f09537fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:16.970 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:16.969+0000 7f09537fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:17.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:29:17.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:29:17.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:29:17.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:29:17.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:29:17.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:17.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:29:17.215 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:29:17.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:17.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:17.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:17.492 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:17.491+0000 7f09537fb4c0 -1 Falling back to public interface 2024-02-05T05:29:18.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:18.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:18.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:29:18.481 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:29:18.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:18.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:18.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:19.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:19.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:19.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:29:19.756 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:29:19.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:19.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:19.828 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:19.827+0000 7f09537fb4c0 -1 osd.1 34 log_to_monitors true 2024-02-05T05:29:20.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:21.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:21.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:21.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:29:21.036 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:29:21.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:21.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/1321288485,v1:127.0.0.1:6811/1321288485] [v2:127.0.0.1:6812/1321288485,v1:127.0.0.1:6813/1321288485] exists,up 86933c9f-8c29-4504-bca3-83bbdb48f121 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:29:21.306 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:29:21.307 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:29:21.307 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:29:21.307 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:29:21.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:29:21.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:29:21.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:29:21.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:29:21.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:29:21.392 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:29:21.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:29:21.657 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:29:21.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:29:21.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:21.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:29:21.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149706 2024-02-05T05:29:21.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149706 2024-02-05T05:29:21.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149706' 2024-02-05T05:29:21.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:21.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:29:21.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757250 2024-02-05T05:29:21.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757250 2024-02-05T05:29:21.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149706 1-163208757250' 2024-02-05T05:29:21.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:21.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149706 2024-02-05T05:29:21.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:21.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:29:21.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149706 2024-02-05T05:29:21.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:21.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149706 2024-02-05T05:29:21.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149706' 2024-02-05T05:29:21.866 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 111669149706 2024-02-05T05:29:21.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:22.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149706 2024-02-05T05:29:22.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:29:23.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:29:23.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:23.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149706 -lt 111669149706 2024-02-05T05:29:23.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:23.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-163208757250 2024-02-05T05:29:23.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:23.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:29:23.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-163208757250 2024-02-05T05:29:23.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757250 2024-02-05T05:29:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-02-05T05:29:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 163208757250 2024-02-05T05:29:23.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:29:23.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757250 2024-02-05T05:29:23.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:29:23.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:23.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:24.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:29:24.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:29:24.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:29:24.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:29:24.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:29:24.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:29:24.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:29:24.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:29:24.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:29:24.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:29:24.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:24.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T05:29:24.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:24.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:25.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:25.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-02-05T05:29:25.415 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.3 on osd.1 to repair 2024-02-05T05:29:25.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:25.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T05:29:25.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:25.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:29:25.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:29:25.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:25.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:25.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:25.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:25.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:25.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:25.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:57.940253+0000 '>' 2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:25.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:26.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:26.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:26.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:26.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:26.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:26.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:26.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:57.940253+0000 '>' 2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:26.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:27.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:27.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:27.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:27.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:27.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:27.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:27.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:28.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:57.940253+0000 '>' 2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:28.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:29.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:29.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:29.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:29.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:29.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:29.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:29.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:29.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:28:57.940253+0000 '>' 2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:29.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:29:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:29:30.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:29:30.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:29:30.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:29:30.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:29:30.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:29:30.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:29:25.747447+0000 '>' 2024-02-05T05:28:57.940253+0000 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:29:30.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:29:30.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:29:30.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:29:30.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:30.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:30.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:30.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:30.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:31.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:31.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-02-05T05:29:31.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:29:31.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:29:31.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:29:31.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:29:31.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:31.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-02-05T05:29:32.167 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-02-05T05:29:32.167 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T05:29:32.167 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T05:29:32.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:32.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:32.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:32.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:29:32.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:29:32.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:29:32.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:29:32.438 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:29:32.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:29:32.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:29:32.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:29:32.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:29:32.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:29:32.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:29:32.465 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:32.464+0000 7f7a167ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:32.486 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:32.485+0000 7f7a167ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:32.487 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:32.486+0000 7f7a167ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:29:32.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:32.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:32.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:33.008 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:33.007+0000 7f7a167ad4c0 -1 Falling back to public interface 2024-02-05T05:29:33.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:33.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:33.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:29:33.980 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:29:33.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:33.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:34.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:29:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:29:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:35.288 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:35.287+0000 7f7a167ad4c0 -1 osd.1 39 log_to_monitors true 2024-02-05T05:29:35.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:36.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:29:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:29:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:29:36.850 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/1579290738,v1:127.0.0.1:6811/1579290738] [v2:127.0.0.1:6812/1579290738,v1:127.0.0.1:6813/1579290738] exists,up 86933c9f-8c29-4504-bca3-83bbdb48f121 2024-02-05T05:29:36.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:29:36.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:29:36.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:29:36.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:29:36.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:29:36.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:29:36.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:29:36.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:29:36.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:29:37.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:29:37.202 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:29:37.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:29:37.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:37.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:29:37.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149710 2024-02-05T05:29:37.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149710 2024-02-05T05:29:37.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149710' 2024-02-05T05:29:37.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:29:37.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:29:37.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-02-05T05:29:37.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-02-05T05:29:37.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149710 1-184683593730' 2024-02-05T05:29:37.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:37.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149710 2024-02-05T05:29:37.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:37.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:29:37.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149710 2024-02-05T05:29:37.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:37.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149710 2024-02-05T05:29:37.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149710' 2024-02-05T05:29:37.410 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 111669149710 2024-02-05T05:29:37.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:37.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149708 -lt 111669149710 2024-02-05T05:29:37.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:29:38.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:29:38.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:29:38.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149710 -lt 111669149710 2024-02-05T05:29:38.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:29:38.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593730 2024-02-05T05:29:38.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:29:38.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:29:38.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593730 2024-02-05T05:29:38.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:29:38.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-02-05T05:29:38.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-02-05T05:29:38.951 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 184683593730 2024-02-05T05:29:38.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:29:39.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-02-05T05:29:39.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:29:39.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:39.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:29:39.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:29:39.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:29:39.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:29:39.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:29:39.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:29:40.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:29:40.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:29:40.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:29:40.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T05:29:40.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T05:29:40.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:29:40.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:29:40.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:29:40.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:29:40.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:40.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:40.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:40.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:40.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:40.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:40.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:29:40.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:29:40.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:29:40.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:29:40.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:29:40.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:29:40.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:40.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:29:40.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:29:40.284 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:40.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:29:40.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:29:40.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:29:40.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:29:40.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:40.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:40.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:29:40.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:29:40.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:29:40.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:29:40.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:29:40.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:29:40.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:29:40.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:29:40.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:29:40.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:29:40.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:29:40.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:29:40.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:29:40.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:29:40.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:40.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:29:40.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:29:40.307 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:29:40.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:29:40.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:29:40.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:29:40.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:29:40.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:40.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:40.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:29:40.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:29:40.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:29:40.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:29:40.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:29:40.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:40.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:40.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:29:40.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:29:40.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:29:40.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:29:40.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:29:40.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:29:40.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:29:40.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:29:40.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:40.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:40.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:40.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:40.693 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:40.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:40.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:29:40.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:29:40.861 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:29:40.862 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:29:40.862 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:40.862 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:40.862 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:29:40.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:29:40.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:29:40.945 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:29:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:40.946 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:29:40.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:29:40.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:29:41.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-02-05T05:29:41.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:29:41.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:29:41.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:29:41.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:29:41.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:29:41.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:29:41.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:29:41.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:41.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:41.828 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:41.828 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:41.829 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:41.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:41.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:29:41.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:29:41.856 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:29:41.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:41.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:29:41.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 13a6ee52-551c-4384-a325-a38a39f705c6' 2024-02-05T05:29:41.864 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:29:41.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:29:41.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFcsBlq3pSNBAAjhx7yQ/Zv3u7sGWYOG/6tA== 2024-02-05T05:29:41.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFcsBlq3pSNBAAjhx7yQ/Zv3u7sGWYOG/6tA=="}' 2024-02-05T05:29:41.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13a6ee52-551c-4384-a325-a38a39f705c6 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:29:42.084 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:29:42.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:29:42.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFcsBlq3pSNBAAjhx7yQ/Zv3u7sGWYOG/6tA== --osd-uuid 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:29:42.123 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:42.121+0000 7f9b7e39b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:42.135 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:42.134+0000 7f9b7e39b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:42.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:42.135+0000 7f9b7e39b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:42.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:42.135+0000 7f9b7e39b4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:29:42.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:42.135+0000 7f9b7e39b4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:29:42.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:42.135+0000 7f9b7e39b4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:29:42.227 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:42.226+0000 7f9b7e39b4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:29:47.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:29:47.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:29:47.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:29:47.755 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:29:47.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:29:48.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:29:48.103 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:29:48.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:29:48.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:29:48.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:29:48.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:29:48.128 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:48.127+0000 7f021fdbc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:48.130 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:48.129+0000 7f021fdbc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:48.131 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:48.130+0000 7f021fdbc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:29:48.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:29:48.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:29:48.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:29:48.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:29:48.371 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:29:48.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:29:48.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:48.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:29:48.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:48.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:48.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:48.651+0000 7f021fdbc4c0 -1 Falling back to public interface 2024-02-05T05:29:49.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:49.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:49.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:29:49.638 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:29:49.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:49.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:49.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:50.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:50.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:50.914 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:29:50.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:29:50.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:50.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:51.008 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:29:51.007+0000 7f021fdbc4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:29:51.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:52.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:52.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:52.189 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:29:52.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:29:52.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:52.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:52.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:53.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:53.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:53.467 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:29:53.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:29:53.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:53.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:53.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:54.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:54.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:54.821 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:29:54.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:29:54.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:54.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:55.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:56.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:56.098 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:29:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:29:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:56.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:56.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:57.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:57.377 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:29:57.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:57.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:29:57.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:57.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:57.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:29:58.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:29:58.654 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:29:58.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:29:58.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:29:58.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:29:58.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:29:59.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:00.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:00.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:00.058 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:30:00.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:30:00.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:00.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:00.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:01.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:01.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:01.338 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:30:01.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:30:01.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:01.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:01.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:02.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:02.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:02.618 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:30:02.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:30:02.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:02.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:02.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:03.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:03.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:03.897 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:30:03.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:30:03.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:03.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:04.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:05.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:05.192 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:30:05.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:05.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:30:05.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:05.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:05.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:06.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:06.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:06.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:30:06.472 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:30:06.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:06.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:06.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:07.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:07.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:07.833 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:30:07.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:30:07.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:07.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:08.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:09.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:09.286 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:30:09.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:09.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:30:09.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:09.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:10.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:11.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:11.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:11.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:30:11.040 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:30:11.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:11.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:11.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:12.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:12.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:12.719 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:30:12.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:30:12.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:12.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:13.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:14.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:14.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:14.514 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:30:14.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:30:14.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:14.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:30:14.784 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4007217094,v1:127.0.0.1:6803/4007217094] [v2:127.0.0.1:6804/4007217094,v1:127.0.0.1:6805/4007217094] exists,up 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:30:14.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:30:14.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:30:14.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:30:14.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:30:14.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:30:14.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cb08c800-4568-4dfa-a2f0-10061c73fc58' 2024-02-05T05:30:14.791 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:30:14.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:30:14.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmcsBlni71LxAAbrMJ+nZ3dTWq3GqCz8WLlA== 2024-02-05T05:30:14.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmcsBlni71LxAAbrMJ+nZ3dTWq3GqCz8WLlA=="}' 2024-02-05T05:30:14.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb08c800-4568-4dfa-a2f0-10061c73fc58 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:30:15.173 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:30:15.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:30:15.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmcsBlni71LxAAbrMJ+nZ3dTWq3GqCz8WLlA== --osd-uuid cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:30:15.212 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:15.211+0000 7f12d2e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:15.257 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:15.256+0000 7f12d2e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:15.258 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:15.257+0000 7f12d2e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:15.259 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:15.257+0000 7f12d2e2a4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:30:15.259 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:15.257+0000 7f12d2e2a4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:30:15.259 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:15.258+0000 7f12d2e2a4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:30:15.324 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:15.323+0000 7f12d2e2a4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:30:21.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:30:21.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:30:21.123 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:30:21.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:30:21.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:30:21.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:30:21.467 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:30:21.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:30:21.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:30:21.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:30:21.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:30:21.493 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:21.492+0000 7f516c4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:21.495 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:21.494+0000 7f516c4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:21.496 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:21.495+0000 7f516c4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:21.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:30:21.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:30:21.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:30:21.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:30:21.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:30:21.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:21.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:30:21.736 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:30:21.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:21.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:22.013 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:22.012+0000 7f516c4144c0 -1 Falling back to public interface 2024-02-05T05:30:22.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:23.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:23.055 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:30:23.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:23.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:30:23.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:23.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:23.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:24.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:24.330 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:30:24.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:24.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:30:24.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:24.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:24.440 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:24.439+0000 7f516c4144c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:30:24.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:25.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:25.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:25.634 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:30:25.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:30:25.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:25.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:25.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:26.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:26.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:26.907 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:30:26.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:30:26.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:26.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:27.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:28.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:28.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:28.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:30:28.287 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:30:28.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:28.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:28.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:29.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:29.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:29.568 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:30:29.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:30:29.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:29.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:29.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:30.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:30.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:30.871 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:30:30.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:30:30.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:30.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:31.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:32.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:32.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:32.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:30:32.253 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:30:32.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:32.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:32.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:33.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:33.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:33.574 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:30:33.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:30:33.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:33.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:33.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:34.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:34.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:34.895 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:30:34.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:30:34.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:34.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:35.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:36.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:36.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:36.251 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:30:36.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:30:36.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:36.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:36.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:37.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:37.873 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:30:37.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:37.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:30:37.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:37.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:38.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:39.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:39.285 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:30:39.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:39.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:30:39.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:39.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:39.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:40.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:40.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:40.983 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:30:40.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:30:40.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:40.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:41.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:42.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:42.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:42.545 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:30:42.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:30:42.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:42.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:43.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:44.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:44.192 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:30:44.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:44.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:30:44.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:44.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:44.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:45.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:45.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:45.567 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:30:45.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:30:45.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:45.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:45.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:46.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:46.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:46.919 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:30:46.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:30:46.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:46.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:30:47.197 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/914100379,v1:127.0.0.1:6811/914100379] [v2:127.0.0.1:6812/914100379,v1:127.0.0.1:6813/914100379] exists,up cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:30:47.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:30:47.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:30:47.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6fcbf89c-15c5-4fb8-b132-14e868aa3594 2024-02-05T05:30:47.204 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 6fcbf89c-15c5-4fb8-b132-14e868aa3594 2024-02-05T05:30:47.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6fcbf89c-15c5-4fb8-b132-14e868aa3594' 2024-02-05T05:30:47.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:30:47.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHcsBlx1sADRAANLr2/a4L9wKVzwgST9sYMg== 2024-02-05T05:30:47.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHcsBlx1sADRAANLr2/a4L9wKVzwgST9sYMg=="}' 2024-02-05T05:30:47.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6fcbf89c-15c5-4fb8-b132-14e868aa3594 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:30:47.571 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:30:47.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:30:47.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHcsBlx1sADRAANLr2/a4L9wKVzwgST9sYMg== --osd-uuid 6fcbf89c-15c5-4fb8-b132-14e868aa3594 2024-02-05T05:30:47.612 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:47.611+0000 7f41cae164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:47.680 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:47.679+0000 7f41cae164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:47.681 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:47.680+0000 7f41cae164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:47.682 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:47.681+0000 7f41cae164c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:30:47.682 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:47.681+0000 7f41cae164c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:30:47.682 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:47.681+0000 7f41cae164c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:30:47.747 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:47.746+0000 7f41cae164c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:30:53.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:30:53.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:30:53.513 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:30:53.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:30:53.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:30:53.860 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:30:53.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:30:53.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:30:53.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:30:53.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:30:53.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:30:53.886 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:53.885+0000 7f83f01244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:53.888 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:53.887+0000 7f83f01244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:53.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:53.888+0000 7f83f01244c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:30:54.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:30:54.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:30:54.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:30:54.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:30:54.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:30:54.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:54.155 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:30:54.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:30:54.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:54.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:54.404 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:54.403+0000 7f83f01244c0 -1 Falling back to public interface 2024-02-05T05:30:54.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:55.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:55.422 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:30:55.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:55.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:30:55.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:55.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:55.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:56.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:56.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:56.695 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:30:56.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:30:56.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:56.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:56.722 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:30:56.721+0000 7f83f01244c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:30:57.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:58.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:58.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:58.040 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:30:58.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:30:58.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:58.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:58.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:30:59.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:30:59.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:30:59.321 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:30:59.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:30:59.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:30:59.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:30:59.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:00.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:00.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:00.605 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:31:00.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:31:00.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:00.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:00.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:01.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:01.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:01.883 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:31:01.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:31:01.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:01.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:02.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:03.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:03.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:03.198 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:31:03.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:31:03.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:03.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:03.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:04.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:04.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:04.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:31:04.528 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:31:04.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:04.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:04.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:05.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:05.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:05.809 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:31:05.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:31:05.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:05.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:06.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:07.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:07.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:07.208 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:31:07.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:31:07.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:07.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:07.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:08.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:08.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:08.562 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:31:08.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:31:08.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:08.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:08.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:09.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:09.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:09.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:31:09.848 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:31:09.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:09.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:10.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:11.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:11.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:11.126 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:31:11.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:31:11.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:11.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:11.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:12.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:12.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:12.656 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:31:12.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:31:12.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:12.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:13.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:14.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:14.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:14.115 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:31:14.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:31:14.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:14.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:14.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:15.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:15.632 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:31:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:31:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:16.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:17.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:17.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:17.110 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:31:17.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:31:17.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:17.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:17.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:18.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:18.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:18.414 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:31:18.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:31:18.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:18.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:18.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:19.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:19.916 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:31:19.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:19.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:31:19.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:19.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:20.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:21.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:21.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:21.219 INFO:tasks.workunit.client.0.smithi188.stdout:20 2024-02-05T05:31:21.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:31:21.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:21.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:31:21.491 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1469250539,v1:127.0.0.1:6819/1469250539] [v2:127.0.0.1:6820/1469250539,v1:127.0.0.1:6821/1469250539] exists,up 6fcbf89c-15c5-4fb8-b132-14e868aa3594 2024-02-05T05:31:21.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:31:21.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:31:21.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:31:21.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-02-05T05:31:21.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:31:21.749 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:31:21.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:31:21.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:31:22.147 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:31:22.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:31:23.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:31:24.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-02-05T05:31:24.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T05:31:25.309 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'foo' created 2024-02-05T05:31:25.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:31:26.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-02-05T05:31:26.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:31:26.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:31:26.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:31:26.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:31:26.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-02-05T05:31:26.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:31:26.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:31:27.277 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:31:27.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:31:28.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:31:28.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:31:28.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:31:28.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:31:28.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:31:28.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:31:28.292 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:31:28.292 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:31:28.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:31:28.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:31:28.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:31:28.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:31:28.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:31:28.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:31:28.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:31:28.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:31:28.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:31:28.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:31:28.655 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:31:28.655 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:31:28.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:31:28.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:28.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:31:28.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-02-05T05:31:28.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-02-05T05:31:28.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-02-05T05:31:28.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:28.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:31:29.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705674 2024-02-05T05:31:29.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705674 2024-02-05T05:31:29.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-38654705674' 2024-02-05T05:31:29.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:29.013 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:31:29.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574853 2024-02-05T05:31:29.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574853 2024-02-05T05:31:29.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-38654705674 2-55834574853' 2024-02-05T05:31:29.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:29.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-02-05T05:31:29.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:29.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:31:29.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-02-05T05:31:29.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:29.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-02-05T05:31:29.119 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836498 2024-02-05T05:31:29.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-02-05T05:31:29.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:29.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-02-05T05:31:29.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:31:30.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:31:30.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:30.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836498 2024-02-05T05:31:30.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:30.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705674 2024-02-05T05:31:30.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:30.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:31:30.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705674 2024-02-05T05:31:30.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:30.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705674 2024-02-05T05:31:30.680 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 38654705674 2024-02-05T05:31:30.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2024-02-05T05:31:30.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:31:30.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705674 -lt 38654705674 2024-02-05T05:31:30.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:30.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-55834574853 2024-02-05T05:31:30.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:30.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:31:30.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-55834574853 2024-02-05T05:31:30.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:30.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574853 2024-02-05T05:31:30.948 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 55834574853 2024-02-05T05:31:30.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 55834574853' 2024-02-05T05:31:30.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:31:31.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574853 -lt 55834574853 2024-02-05T05:31:31.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:31:31.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:31.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:31.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:31:31.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:31:31.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:31:31.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:31:31.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:31:31.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:31:31.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:31:31.548 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:31:31.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:31:31.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:31:31.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:31.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:32.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:31:32.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:31:32.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:31:32.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:31:32.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-02-05T05:31:32.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:31:32.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:31:32.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:31:32.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:31:32.143 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:31:32.143 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:31:32.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:31:32.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:31:32.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:31:32.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:31:32.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:31:32.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:31:32.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:31:32.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:31:32.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:31:32.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:31:32.494 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:31:32.494 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:31:32.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:31:32.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:32.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:31:32.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T05:31:32.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T05:31:32.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T05:31:32.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:32.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:31:32.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705676 2024-02-05T05:31:32.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705676 2024-02-05T05:31:32.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-38654705676' 2024-02-05T05:31:32.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:32.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:31:32.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574855 2024-02-05T05:31:32.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574855 2024-02-05T05:31:32.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-38654705676 2-55834574855' 2024-02-05T05:31:32.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:32.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T05:31:32.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:32.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:31:32.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T05:31:32.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:32.803 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836500 2024-02-05T05:31:32.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T05:31:32.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T05:31:32.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:33.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-02-05T05:31:33.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:31:34.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:31:34.071 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:34.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-02-05T05:31:34.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:34.352 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705676 2024-02-05T05:31:34.352 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:34.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:31:34.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705676 2024-02-05T05:31:34.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:34.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705676 2024-02-05T05:31:34.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2024-02-05T05:31:34.355 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 38654705676 2024-02-05T05:31:34.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:31:34.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705676 -lt 38654705676 2024-02-05T05:31:34.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:34.640 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-55834574855 2024-02-05T05:31:34.640 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:34.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:31:34.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-55834574855 2024-02-05T05:31:34.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:34.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574855 2024-02-05T05:31:34.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2024-02-05T05:31:34.643 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 55834574855 2024-02-05T05:31:34.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:31:34.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574855 -lt 55834574855 2024-02-05T05:31:34.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:31:34.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:34.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:31:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:31:35.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:31:35.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:31:35.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:31:35.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:31:35.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:31:35.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:31:35.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:31:35.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:31:35.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:35.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:35.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:31:35.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:31:35.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:31:35.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:31:35.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:31:36.222 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:31:36.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:31:36.733 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:31:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:31:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:31:36.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:31:37.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-02-05T05:31:37.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:31:37.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:31:37.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:31:37.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:31:37.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:31:37.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:31:37.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:31:37.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:31:37.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:39.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:31:39.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:31:39.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:31:39.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:31:39.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:31:39.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:31:39.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:31:39.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:31:39.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:31:39.680 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:39.678+0000 7f052de1c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:39.682 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:39.681+0000 7f052de1c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:39.683 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:39.682+0000 7f052de1c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:39.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:31:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:31:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:31:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:31:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:31:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:31:39.932 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:31:39.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:39.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:31:40.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:40.205 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:40.204+0000 7f052de1c4c0 -1 Falling back to public interface 2024-02-05T05:31:41.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:41.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:41.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:31:41.202 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:31:41.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:41.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:31:41.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:42.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:42.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:42.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:31:42.479 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:31:42.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:42.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:31:42.568 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:42.566+0000 7f052de1c4c0 -1 osd.1 32 log_to_monitors true 2024-02-05T05:31:42.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:43.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:43.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:43.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:31:43.780 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:31:43.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:43.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:31:44.185 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 33 last_clean_interval [9,32) [v2:127.0.0.1:6810/4043345630,v1:127.0.0.1:6811/4043345630] [v2:127.0.0.1:6812/4043345630,v1:127.0.0.1:6813/4043345630] exists,up cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:31:44.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:31:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:31:44.187 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:31:44.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:31:44.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:31:44.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:31:44.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:31:44.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:31:44.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:31:44.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:31:44.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:31:44.272 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:31:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:31:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:31:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:31:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:31:44.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:44.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:31:44.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-05T05:31:44.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-05T05:31:44.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-05T05:31:44.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:44.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:31:44.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789954 2024-02-05T05:31:44.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789954 2024-02-05T05:31:44.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-158913789954' 2024-02-05T05:31:44.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:44.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:31:44.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574858 2024-02-05T05:31:44.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574858 2024-02-05T05:31:44.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-158913789954 2-55834574858' 2024-02-05T05:31:44.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:44.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-05T05:31:44.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:44.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:31:44.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-05T05:31:44.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:44.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-05T05:31:44.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-05T05:31:44.872 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836504 2024-02-05T05:31:44.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:45.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-02-05T05:31:45.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:31:46.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:31:46.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:46.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-02-05T05:31:46.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:46.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-158913789954 2024-02-05T05:31:46.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:46.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:31:46.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-158913789954 2024-02-05T05:31:46.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:46.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789954 2024-02-05T05:31:46.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 158913789954' 2024-02-05T05:31:46.421 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 158913789954 2024-02-05T05:31:46.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:31:46.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789954 -lt 158913789954 2024-02-05T05:31:46.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:46.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-55834574858 2024-02-05T05:31:46.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:46.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:31:46.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-55834574858 2024-02-05T05:31:46.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:46.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574858 2024-02-05T05:31:46.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2024-02-05T05:31:46.692 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 55834574858 2024-02-05T05:31:46.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:31:46.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574858 -lt 55834574858 2024-02-05T05:31:46.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:31:46.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:46.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:31:47.291 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:31:47.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:31:47.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:31:47.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:47.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:47.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-02-05T05:31:47.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-02-05T05:31:47.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:31:47.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-02-05T05:31:47.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:31:47.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:31:47.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:31:47.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:31:47.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:31:47.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:31:47.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:31:47.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:31:47.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:31:47.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:31:48.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:31:48.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:31:48.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:48.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:31:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:31:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:31:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:31:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-02-05T05:31:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-02-05T05:31:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:31:48.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:31:48.944 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:31:48.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:31:49.362 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:31:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:31:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:31:49.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:31:49.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-02-05T05:31:49.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:31:49.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:31:49.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-02-05T05:31:49.653 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.8499e-05 s, 29.9 MB/s 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:31:49.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:31:49.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:31:49.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:31:49.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:31:49.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:31:49.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:31:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:31:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:31:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:31:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:31:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:31:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:31:52.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:31:52.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:31:52.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:31:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:31:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:31:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:31:52.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:31:52.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:31:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:31:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:31:52.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:31:52.396 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:52.395+0000 7f35c14384c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:52.398 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:52.397+0000 7f35c14384c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:52.399 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:52.398+0000 7f35c14384c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:31:52.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:31:52.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:31:52.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:31:52.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:31:52.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:31:52.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:52.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:31:52.649 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:31:52.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:52.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:31:52.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:52.920 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:52.919+0000 7f35c14384c0 -1 Falling back to public interface 2024-02-05T05:31:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:31:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:31:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:31:54.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:55.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:55.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:55.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:31:55.195 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:31:55.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:55.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:31:55.276 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:31:55.274+0000 7f35c14384c0 -1 osd.0 41 log_to_monitors true 2024-02-05T05:31:55.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:31:56.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:31:56.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:31:56.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:31:56.476 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:31:56.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:31:56.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:31:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 46 up_thru 46 down_at 42 last_clean_interval [5,41) [v2:127.0.0.1:6802/767677939,v1:127.0.0.1:6803/767677939] [v2:127.0.0.1:6804/767677939,v1:127.0.0.1:6805/767677939] exists,up 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:31:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:31:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:31:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:31:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:31:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:31:56.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:31:56.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:31:56.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:31:56.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:31:56.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:31:56.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:31:56.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:31:57.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:31:57.212 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:31:57.212 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:31:57.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:31:57.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:57.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:31:57.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-02-05T05:31:57.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-02-05T05:31:57.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495618' 2024-02-05T05:31:57.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:57.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:31:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789957 2024-02-05T05:31:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789957 2024-02-05T05:31:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495618 1-158913789957' 2024-02-05T05:31:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:31:57.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:31:57.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574862 2024-02-05T05:31:57.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574862 2024-02-05T05:31:57.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495618 1-158913789957 2-55834574862' 2024-02-05T05:31:57.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:57.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-197568495618 2024-02-05T05:31:57.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:57.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:31:57.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-197568495618 2024-02-05T05:31:57.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:57.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-02-05T05:31:57.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 197568495618' 2024-02-05T05:31:57.525 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 197568495618 2024-02-05T05:31:57.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:57.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 197568495618 2024-02-05T05:31:57.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:31:58.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:31:58.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:31:59.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-02-05T05:31:59.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:59.071 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-158913789957 2024-02-05T05:31:59.071 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:59.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:31:59.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-158913789957 2024-02-05T05:31:59.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:59.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789957 2024-02-05T05:31:59.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 158913789957' 2024-02-05T05:31:59.073 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 158913789957 2024-02-05T05:31:59.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:31:59.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789957 -lt 158913789957 2024-02-05T05:31:59.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:31:59.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-55834574862 2024-02-05T05:31:59.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:31:59.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:31:59.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-55834574862 2024-02-05T05:31:59.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:31:59.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574862 2024-02-05T05:31:59.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 55834574862' 2024-02-05T05:31:59.348 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 55834574862 2024-02-05T05:31:59.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:31:59.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574862 -lt 55834574862 2024-02-05T05:31:59.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:31:59.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:31:59.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:31:59.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:31:59.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:31:59.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:31:59.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:31:59.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:31:59.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:31:59.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:31:59.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:00.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:32:00.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:00.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:00.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:00.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:32:00.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:32:00.937 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:32:00.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:32:01.456 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:32:01.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:32:01.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:32:01.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:32:01.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:01.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:01.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:32:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:32:01.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:01.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:32:01.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-02-05T05:32:03.533 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-02-05T05:32:04.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:04.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:32:04.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:32:04.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:04.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:04.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:32:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:32:04.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:04.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:04.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:04.085 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:04.084+0000 7fa7017174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:04.086+0000 7fa7017174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:04.088 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:04.087+0000 7fa7017174c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:04.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:32:04.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:04.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:32:04.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:04.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:04.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:04.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:04.338 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:32:04.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:04.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:04.611 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:04.610+0000 7fa7017174c0 -1 Falling back to public interface 2024-02-05T05:32:04.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:05.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:05.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:05.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:05.670 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:05.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:05.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:06.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:06.876 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:06.875+0000 7fa7017174c0 -1 osd.1 50 log_to_monitors true 2024-02-05T05:32:07.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:07.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:07.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:07.005 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:32:07.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:07.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:07.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:08.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:08.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:08.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:08.286 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:32:08.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:08.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:08.644 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 55 up_thru 55 down_at 51 last_clean_interval [37,50) [v2:127.0.0.1:6810/3800366138,v1:127.0.0.1:6811/3800366138] [v2:127.0.0.1:6812/3800366138,v1:127.0.0.1:6813/3800366138] exists,up cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:32:08.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:08.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:08.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:08.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:08.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:08.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:08.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:08.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:09.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:09.014 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:09.014 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:32:09.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:09.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:09.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:09.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495621 2024-02-05T05:32:09.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495621 2024-02-05T05:32:09.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495621' 2024-02-05T05:32:09.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:09.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:09.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201282 2024-02-05T05:32:09.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201282 2024-02-05T05:32:09.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495621 1-236223201282' 2024-02-05T05:32:09.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:09.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:09.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574865 2024-02-05T05:32:09.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574865 2024-02-05T05:32:09.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-197568495621 1-236223201282 2-55834574865' 2024-02-05T05:32:09.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:09.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-197568495621 2024-02-05T05:32:09.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:09.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:09.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-197568495621 2024-02-05T05:32:09.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:09.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495621 2024-02-05T05:32:09.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 197568495621' 2024-02-05T05:32:09.326 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 197568495621 2024-02-05T05:32:09.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:09.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495620 -lt 197568495621 2024-02-05T05:32:09.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:10.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:32:10.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:10.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495621 -lt 197568495621 2024-02-05T05:32:10.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:10.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201282 2024-02-05T05:32:10.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:10.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:10.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201282 2024-02-05T05:32:10.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:10.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201282 2024-02-05T05:32:10.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201282' 2024-02-05T05:32:10.873 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 236223201282 2024-02-05T05:32:10.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:11.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201282 -lt 236223201282 2024-02-05T05:32:11.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:11.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-55834574865 2024-02-05T05:32:11.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:11.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:11.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-55834574865 2024-02-05T05:32:11.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:11.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574865 2024-02-05T05:32:11.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 55834574865' 2024-02-05T05:32:11.145 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 55834574865 2024-02-05T05:32:11.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:11.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574865 -lt 55834574865 2024-02-05T05:32:11.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:11.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:11.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:11.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:12.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:32:12.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:12.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:12.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:12.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:32:12.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:32:12.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:32:12.712 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:32:12.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:32:13.122 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:32:13.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:32:13.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:32:13.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:32:13.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-02-05T05:32:13.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:32:13.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:32:13.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-02-05T05:32:13.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-02-05T05:32:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T05:32:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:32:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:32:13.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:14.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:14.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:32:14.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:14.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:14.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:32:14.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:14.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:32:14.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:32:16.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:32:16.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:16.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:32:16.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:16.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:32:16.034 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:32:16.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:16.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:32:16.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:32:16.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:16.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:16.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:16.059+0000 7f2aeee154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:16.062+0000 7f2aeee154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:16.064 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:16.063+0000 7f2aeee154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:16.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:32:16.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:16.588 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:16.587+0000 7f2aeee154c0 -1 Falling back to public interface 2024-02-05T05:32:17.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:17.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:17.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:17.575 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:17.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:17.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:32:17.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:18.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:18.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:18.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:18.846 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:32:18.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:18.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:32:18.903 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:18.902+0000 7f2aeee154c0 -1 osd.0 59 log_to_monitors true 2024-02-05T05:32:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:20.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:20.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:20.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:20.119 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:32:20.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:20.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:32:20.567 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 63 up_thru 55 down_at 60 last_clean_interval [46,59) [v2:127.0.0.1:6802/2683592564,v1:127.0.0.1:6803/2683592564] [v2:127.0.0.1:6804/2683592564,v1:127.0.0.1:6805/2683592564] exists,up 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:32:20.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:20.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:20.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:20.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:20.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:20.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:20.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:20.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:20.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:20.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:20.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:20.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:21.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:21.022 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:21.022 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:32:21.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:21.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:21.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:21.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939650 2024-02-05T05:32:21.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939650 2024-02-05T05:32:21.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939650' 2024-02-05T05:32:21.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:21.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:21.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201285 2024-02-05T05:32:21.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201285 2024-02-05T05:32:21.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939650 1-236223201285' 2024-02-05T05:32:21.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:21.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:21.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574869 2024-02-05T05:32:21.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574869 2024-02-05T05:32:21.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939650 1-236223201285 2-55834574869' 2024-02-05T05:32:21.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:21.357 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-270582939650 2024-02-05T05:32:21.357 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:21.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:21.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-270582939650 2024-02-05T05:32:21.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:21.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939650 2024-02-05T05:32:21.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 270582939650' 2024-02-05T05:32:21.359 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 270582939650 2024-02-05T05:32:21.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:21.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 270582939650 2024-02-05T05:32:21.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:22.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:32:22.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:22.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939650 -lt 270582939650 2024-02-05T05:32:22.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:22.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201285 2024-02-05T05:32:22.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:22.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:22.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201285 2024-02-05T05:32:22.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:22.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201285 2024-02-05T05:32:22.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201285' 2024-02-05T05:32:22.898 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 236223201285 2024-02-05T05:32:22.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:23.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201285 -lt 236223201285 2024-02-05T05:32:23.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:23.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-55834574869 2024-02-05T05:32:23.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:23.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:23.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-55834574869 2024-02-05T05:32:23.167 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:23.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574869 2024-02-05T05:32:23.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 55834574869' 2024-02-05T05:32:23.168 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 55834574869 2024-02-05T05:32:23.168 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:23.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574869 -lt 55834574869 2024-02-05T05:32:23.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:23.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:23.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:23.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:32:23.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:23.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:23.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:23.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:23.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:23.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:23.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:24.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:32:24.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:24.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:24.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:24.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:32:24.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:24.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:24.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:24.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:24.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:24.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:24.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:32:24.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:24.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:24.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:32:24.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:24.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:32:24.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:32:27.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:32:27.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:27.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:27.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:32:27.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:27.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:27.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:27.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:27.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:32:27.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:32:27.032 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.2 2024-02-05T05:32:27.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:27.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:32:27.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:32:27.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:27.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:27.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:27.058 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:27.056+0000 7f3d0ee1c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:27.060 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:27.059+0000 7f3d0ee1c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:27.061 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:27.060+0000 7f3d0ee1c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:27.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:32:27.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:27.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:32:27.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:27.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:27.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:27.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:27.312 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:32:27.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:27.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:27.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:27.578 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:27.577+0000 7f3d0ee1c4c0 -1 Falling back to public interface 2024-02-05T05:32:28.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:28.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:28.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:28.578 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:28.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:28.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:28.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:29.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:29.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:29.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:29.850 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:32:29.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:29.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:30.061 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:30.059+0000 7f3d0ee1c4c0 -1 osd.2 66 log_to_monitors true 2024-02-05T05:32:30.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:31.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:31.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:31.177 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:32:31.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:31.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:31.593 INFO:tasks.workunit.client.0.smithi188.stderr:osd.2 up in weight 1 up_from 71 up_thru 71 down_at 67 last_clean_interval [13,66) [v2:127.0.0.1:6818/2699402301,v1:127.0.0.1:6819/2699402301] [v2:127.0.0.1:6820/2699402301,v1:127.0.0.1:6821/2699402301] exists,up 6fcbf89c-15c5-4fb8-b132-14e868aa3594 2024-02-05T05:32:31.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:31.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:31.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:31.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:31.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:31.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:31.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:31.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:32.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:32.006 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:32.006 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:32:32.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:32.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:32.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:32.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939653 2024-02-05T05:32:32.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939653 2024-02-05T05:32:32.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939653' 2024-02-05T05:32:32.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:32.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:32.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201289 2024-02-05T05:32:32.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201289 2024-02-05T05:32:32.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939653 1-236223201289' 2024-02-05T05:32:32.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:32.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:32.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=304942678018 2024-02-05T05:32:32.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 304942678018 2024-02-05T05:32:32.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939653 1-236223201289 2-304942678018' 2024-02-05T05:32:32.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:32.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-270582939653 2024-02-05T05:32:32.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:32.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:32.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-270582939653 2024-02-05T05:32:32.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:32.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939653 2024-02-05T05:32:32.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 270582939653' 2024-02-05T05:32:32.317 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 270582939653 2024-02-05T05:32:32.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:32.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939652 -lt 270582939653 2024-02-05T05:32:32.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:33.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:32:33.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:33.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939652 -lt 270582939653 2024-02-05T05:32:33.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:34.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:32:34.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:35.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939653 -lt 270582939653 2024-02-05T05:32:35.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:35.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201289 2024-02-05T05:32:35.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:35.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:35.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201289 2024-02-05T05:32:35.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:35.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201289 2024-02-05T05:32:35.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201289' 2024-02-05T05:32:35.142 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 236223201289 2024-02-05T05:32:35.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:35.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201289 -lt 236223201289 2024-02-05T05:32:35.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:35.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-304942678018 2024-02-05T05:32:35.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:35.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:35.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-304942678018 2024-02-05T05:32:35.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:35.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=304942678018 2024-02-05T05:32:35.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 304942678018' 2024-02-05T05:32:35.412 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 304942678018 2024-02-05T05:32:35.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:35.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 304942678018 -lt 304942678018 2024-02-05T05:32:35.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:35.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:35.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:36.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:32:36.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:36.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:36.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:36.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:36.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:36.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:36.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:36.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:32:36.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:36.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:36.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:36.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:32:36.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:36.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:36.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:32:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:32:36.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:32:36.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:32:39.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:32:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:39.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:39.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:39.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:39.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:39.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:32:39.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:32:39.078 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.2 2024-02-05T05:32:39.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:39.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:32:39.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:32:39.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:39.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:39.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:39.105 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:39.104+0000 7fbeb614a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:39.107 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:39.106+0000 7fbeb614a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:39.109 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:39.107+0000 7fbeb614a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:39.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:32:39.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:39.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:32:39.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:39.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:39.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:39.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:39.352 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:32:39.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:39.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:39.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:39.633 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:39.632+0000 7fbeb614a4c0 -1 Falling back to public interface 2024-02-05T05:32:40.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:40.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:40.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:40.620 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:40.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:40.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:40.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:41.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:41.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:41.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:41.898 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:32:41.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:41.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:42.082 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:42.081+0000 7fbeb614a4c0 -1 osd.2 73 log_to_monitors true 2024-02-05T05:32:42.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:32:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:osd.2 up in weight 1 up_from 78 up_thru 78 down_at 74 last_clean_interval [71,73) [v2:127.0.0.1:6818/2346986814,v1:127.0.0.1:6819/2346986814] [v2:127.0.0.1:6820/2346986814,v1:127.0.0.1:6821/2346986814] exists,up 6fcbf89c-15c5-4fb8-b132-14e868aa3594 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:43.515 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:43.516 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:43.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:43.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:43.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:43.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:43.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:43.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:43.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:43.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:43.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:43.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:43.938 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:43.938 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:32:43.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:43.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:43.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:44.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939656 2024-02-05T05:32:44.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939656 2024-02-05T05:32:44.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939656' 2024-02-05T05:32:44.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:44.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:44.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201292 2024-02-05T05:32:44.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201292 2024-02-05T05:32:44.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939656 1-236223201292' 2024-02-05T05:32:44.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:44.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:44.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-02-05T05:32:44.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-02-05T05:32:44.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939656 1-236223201292 2-335007449090' 2024-02-05T05:32:44.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:44.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-270582939656 2024-02-05T05:32:44.281 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:44.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:44.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-270582939656 2024-02-05T05:32:44.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:44.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939656 2024-02-05T05:32:44.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 270582939656' 2024-02-05T05:32:44.284 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 270582939656 2024-02-05T05:32:44.284 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:44.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939655 -lt 270582939656 2024-02-05T05:32:44.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:45.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:32:45.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:45.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939655 -lt 270582939656 2024-02-05T05:32:45.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:46.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:32:46.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:47.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939657 -lt 270582939656 2024-02-05T05:32:47.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:47.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-236223201292 2024-02-05T05:32:47.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:47.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:47.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-236223201292 2024-02-05T05:32:47.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:47.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201292 2024-02-05T05:32:47.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 236223201292' 2024-02-05T05:32:47.112 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 236223201292 2024-02-05T05:32:47.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:47.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201292 -lt 236223201292 2024-02-05T05:32:47.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:47.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449090 2024-02-05T05:32:47.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:47.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:47.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449090 2024-02-05T05:32:47.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:47.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-02-05T05:32:47.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449090' 2024-02-05T05:32:47.383 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 335007449090 2024-02-05T05:32:47.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:47.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449090 -lt 335007449090 2024-02-05T05:32:47.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:47.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:47.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:47.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:32:47.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:47.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:47.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:47.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:47.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:47.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:47.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:32:48.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:32:48.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:32:48.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:48.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:48.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:32:48.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:32:48.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:32:48.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:32:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:32:49.037 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:32:49.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:32:49.422 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:32:49.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:32:49.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:32:49.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:32:49.673 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-02-05T05:32:49.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:32:49.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:32:49.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-02-05T05:32:49.676 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records in 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records out 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.8927e-05 s, 59.4 MB/s 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:32:49.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:32:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:32:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:32:49.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:32:49.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:32:49.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:32:49.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:32:49.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:32:49.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:32:49.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:32:49.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:32:49.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:32:52.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:32:52.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:32:52.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:32:52.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:32:52.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:32:52.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:32:52.216 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:32:52.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:32:52.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:32:52.219 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:32:52.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:32:52.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:32:52.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:32:52.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:32:52.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:32:52.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:32:52.244 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:52.243+0000 7fde79caf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:52.246 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:52.245+0000 7fde79caf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:52.247 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:52.246+0000 7fde79caf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:32:52.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:32:52.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:32:52.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:32:52.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:32:52.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:32:52.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:52.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:32:52.494 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:32:52.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:52.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:52.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:52.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:52.773+0000 7fde79caf4c0 -1 Falling back to public interface 2024-02-05T05:32:53.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:53.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:53.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:32:53.759 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:53.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:53.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:54.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:55.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:55.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:55.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:32:55.061 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:32:55.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:55.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:55.153 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:32:55.151+0000 7fde79caf4c0 -1 osd.1 82 log_to_monitors true 2024-02-05T05:32:55.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:32:56.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:32:56.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:32:56.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:32:56.487 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:32:56.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:32:56.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:32:56.766 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 87 up_thru 87 down_at 83 last_clean_interval [55,82) [v2:127.0.0.1:6810/2357433676,v1:127.0.0.1:6811/2357433676] [v2:127.0.0.1:6812/2357433676,v1:127.0.0.1:6813/2357433676] exists,up cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:32:56.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:32:56.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:32:56.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:32:56.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:32:56.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:32:56.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:32:56.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:32:56.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:32:57.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:32:57.121 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:32:57.121 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:32:57.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:32:57.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:57.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:32:57.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939660 2024-02-05T05:32:57.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939660 2024-02-05T05:32:57.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939660' 2024-02-05T05:32:57.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:57.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:32:57.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154754 2024-02-05T05:32:57.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154754 2024-02-05T05:32:57.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939660 1-373662154754' 2024-02-05T05:32:57.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:32:57.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:32:57.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449094 2024-02-05T05:32:57.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449094 2024-02-05T05:32:57.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-270582939660 1-373662154754 2-335007449094' 2024-02-05T05:32:57.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:57.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-270582939660 2024-02-05T05:32:57.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:57.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:32:57.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-270582939660 2024-02-05T05:32:57.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:57.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939660 2024-02-05T05:32:57.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 270582939660' 2024-02-05T05:32:57.435 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 270582939660 2024-02-05T05:32:57.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:57.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939659 -lt 270582939660 2024-02-05T05:32:57.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:32:58.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:32:58.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:32:58.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939660 -lt 270582939660 2024-02-05T05:32:58.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:58.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154754 2024-02-05T05:32:58.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:58.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:32:58.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154754 2024-02-05T05:32:58.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154754 2024-02-05T05:32:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154754' 2024-02-05T05:32:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 373662154754 2024-02-05T05:32:58.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:32:59.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154754 -lt 373662154754 2024-02-05T05:32:59.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:32:59.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449094 2024-02-05T05:32:59.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:32:59.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:32:59.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449094 2024-02-05T05:32:59.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:32:59.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449094 2024-02-05T05:32:59.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449094' 2024-02-05T05:32:59.252 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 335007449094 2024-02-05T05:32:59.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:32:59.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449094 -lt 335007449094 2024-02-05T05:32:59.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:32:59.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:32:59.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:32:59.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:32:59.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:32:59.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:32:59.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:32:59.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:32:59.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:32:59.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:32:59.849 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:00.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:33:00.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:00.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:00.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:00.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:33:00.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:00.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:00.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:33:00.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:33:00.804 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:33:00.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:33:01.299 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:33:01.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:33:01.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:33:01.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:33:01.527 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-02-05T05:33:01.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:33:01.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:33:01.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:01.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:01.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:33:01.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:33:04.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:33:04.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:04.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:04.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:33:04.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:33:04.056 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:33:04.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:04.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:33:04.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:33:04.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:04.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:04.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:04.081+0000 7efc95a234c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:04.084 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:04.083+0000 7efc95a234c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:04.086 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:04.084+0000 7efc95a234c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:33:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:33:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:04.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:33:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:04.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:04.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:04.611 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:04.610+0000 7efc95a234c0 -1 Falling back to public interface 2024-02-05T05:33:05.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:05.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:05.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:05.603 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:05.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:05.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:05.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:06.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:06.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:06.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:06.883 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:33:06.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:06.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:07.055 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:07.053+0000 7efc95a234c0 -1 osd.0 91 log_to_monitors true 2024-02-05T05:33:07.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:08.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:08.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:08.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:08.167 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:33:08.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:08.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:08.512 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 96 up_thru 96 down_at 92 last_clean_interval [63,91) [v2:127.0.0.1:6802/697349558,v1:127.0.0.1:6803/697349558] [v2:127.0.0.1:6804/697349558,v1:127.0.0.1:6805/697349558] exists,up 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:33:08.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:08.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:08.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:08.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:08.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:08.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:08.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:08.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:08.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:08.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:08.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:08.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:33:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:08.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:08.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:09.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860418 2024-02-05T05:33:09.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860418 2024-02-05T05:33:09.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-412316860418' 2024-02-05T05:33:09.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:09.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:09.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=373662154757 2024-02-05T05:33:09.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 373662154757 2024-02-05T05:33:09.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-412316860418 1-373662154757' 2024-02-05T05:33:09.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:09.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449097 2024-02-05T05:33:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449097 2024-02-05T05:33:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-412316860418 1-373662154757 2-335007449097' 2024-02-05T05:33:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:09.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-412316860418 2024-02-05T05:33:09.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:09.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:09.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-412316860418 2024-02-05T05:33:09.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:09.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860418 2024-02-05T05:33:09.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 412316860418' 2024-02-05T05:33:09.386 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 412316860418 2024-02-05T05:33:09.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:09.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 412316860418 2024-02-05T05:33:09.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:10.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:10.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:11.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860418 -lt 412316860418 2024-02-05T05:33:11.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:11.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-373662154757 2024-02-05T05:33:11.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:11.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:11.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-373662154757 2024-02-05T05:33:11.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:11.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=373662154757 2024-02-05T05:33:11.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 373662154757' 2024-02-05T05:33:11.003 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 373662154757 2024-02-05T05:33:11.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:11.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 373662154757 -lt 373662154757 2024-02-05T05:33:11.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:11.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449097 2024-02-05T05:33:11.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:11.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:11.272 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449097 2024-02-05T05:33:11.272 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:11.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449097 2024-02-05T05:33:11.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449097' 2024-02-05T05:33:11.273 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 335007449097 2024-02-05T05:33:11.273 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:11.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449097 -lt 335007449097 2024-02-05T05:33:11.542 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:11.542 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:11.542 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:11.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:12.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:33:12.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:12.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:12.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:12.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:33:12.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:12.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:12.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T05:33:12.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:33:12.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:12.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:12.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:33:12.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:12.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:33:12.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:12.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:12.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:33:12.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:12.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:33:12.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:12.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:12.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:12.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:12.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:12.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:33:14.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:33:14.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:14.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:14.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:33:14.660 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:33:14.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:14.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:33:14.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:33:14.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:14.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:14.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:14.687 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:14.686+0000 7fbda24d84c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:14.690 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:14.689+0000 7fbda24d84c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:14.691 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:14.690+0000 7fbda24d84c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:33:14.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:14.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:15.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:15.212 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:15.211+0000 7fbda24d84c0 -1 Falling back to public interface 2024-02-05T05:33:16.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:16.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:16.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:16.208 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:16.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:16.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:16.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:17.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:17.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:17.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:17.486 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:33:17.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:17.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:17.567 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:17.565+0000 7fbda24d84c0 -1 osd.1 99 log_to_monitors true 2024-02-05T05:33:17.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:18.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:18.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:18.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:18.814 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:33:18.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:18.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 103 up_thru 103 down_at 100 last_clean_interval [87,99) [v2:127.0.0.1:6810/3678526747,v1:127.0.0.1:6811/3678526747] [v2:127.0.0.1:6812/3678526747,v1:127.0.0.1:6813/3678526747] exists,up cb08c800-4568-4dfa-a2f0-10061c73fc58 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:19.298 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:19.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:19.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:19.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:19.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:19.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:19.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:19.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:19.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:19.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:19.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:19.670 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:19.670 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:33:19.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:19.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:19.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:19.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=412316860421 2024-02-05T05:33:19.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 412316860421 2024-02-05T05:33:19.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-412316860421' 2024-02-05T05:33:19.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:19.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:19.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631490 2024-02-05T05:33:19.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631490 2024-02-05T05:33:19.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-412316860421 1-442381631490' 2024-02-05T05:33:19.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:19.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:19.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449100 2024-02-05T05:33:19.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449100 2024-02-05T05:33:19.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-412316860421 1-442381631490 2-335007449100' 2024-02-05T05:33:19.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:19.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-412316860421 2024-02-05T05:33:19.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:19.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:19.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-412316860421 2024-02-05T05:33:19.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:19.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=412316860421 2024-02-05T05:33:19.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 412316860421' 2024-02-05T05:33:19.992 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 412316860421 2024-02-05T05:33:19.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:20.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860421 2024-02-05T05:33:20.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:21.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:21.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:21.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860420 -lt 412316860421 2024-02-05T05:33:21.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:22.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:33:22.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:22.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 412316860421 -lt 412316860421 2024-02-05T05:33:22.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:22.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-442381631490 2024-02-05T05:33:22.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:22.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:22.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-442381631490 2024-02-05T05:33:22.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:22.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631490 2024-02-05T05:33:22.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 442381631490' 2024-02-05T05:33:22.807 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 442381631490 2024-02-05T05:33:22.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:23.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631490 -lt 442381631490 2024-02-05T05:33:23.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:23.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449100 2024-02-05T05:33:23.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:23.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:23.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449100 2024-02-05T05:33:23.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:23.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449100 2024-02-05T05:33:23.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449100' 2024-02-05T05:33:23.076 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 335007449100 2024-02-05T05:33:23.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:23.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449100 -lt 335007449100 2024-02-05T05:33:23.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:23.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:23.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:33:23.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:23.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:23.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:23.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:23.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:23.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:23.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:23.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:33:23.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:23.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:23.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:33:24.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-02-05T05:33:24.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:33:24.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:33:24.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:33:24.702 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:33:24.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:33:25.229 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:33:25.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:33:25.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:33:25.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:33:25.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-02-05T05:33:25.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:33:25.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:33:25.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T05:33:25.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T05:33:25.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-02-05T05:33:25.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:33:25.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:25.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:25.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:33:25.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:25.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:25.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:25.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:33:25.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:25.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:25.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:33:25.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:25.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:33:25.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:33:27.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:33:27.828 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:33:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:27.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:33:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:33:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:27.854 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:27.853+0000 7f25f3c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:27.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:27.855+0000 7f25f3c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:27.857 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:27.856+0000 7f25f3c204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:28.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:28.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:28.379 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:28.378+0000 7f25f3c204c0 -1 Falling back to public interface 2024-02-05T05:33:29.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:29.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:29.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:29.375 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:29.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:29.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:29.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:33:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:30.706 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:30.705+0000 7f25f3c204c0 -1 osd.0 107 log_to_monitors true 2024-02-05T05:33:30.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:31.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:31.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:31.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:31.928 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:33:31.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:31.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:32.201 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 112 up_thru 112 down_at 108 last_clean_interval [96,107) [v2:127.0.0.1:6802/49889541,v1:127.0.0.1:6803/49889541] [v2:127.0.0.1:6804/49889541,v1:127.0.0.1:6805/49889541] exists,up 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:32.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:32.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:32.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:32.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:32.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:32.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:32.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:32.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:32.599 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:32.599 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:33:32.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:32.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:32.599 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:32.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-02-05T05:33:32.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-02-05T05:33:32.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-481036337154' 2024-02-05T05:33:32.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:32.702 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:32.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631493 2024-02-05T05:33:32.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631493 2024-02-05T05:33:32.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-481036337154 1-442381631493' 2024-02-05T05:33:32.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:32.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:32.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449104 2024-02-05T05:33:32.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449104 2024-02-05T05:33:32.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-481036337154 1-442381631493 2-335007449104' 2024-02-05T05:33:32.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:32.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-481036337154 2024-02-05T05:33:32.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:32.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:32.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-481036337154 2024-02-05T05:33:32.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:32.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-02-05T05:33:32.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 481036337154' 2024-02-05T05:33:32.912 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 481036337154 2024-02-05T05:33:32.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:33.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 481036337154 2024-02-05T05:33:33.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:34.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:34.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:34.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337154 -lt 481036337154 2024-02-05T05:33:34.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:34.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-442381631493 2024-02-05T05:33:34.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:34.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:34.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-442381631493 2024-02-05T05:33:34.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:34.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631493 2024-02-05T05:33:34.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 442381631493' 2024-02-05T05:33:34.458 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 442381631493 2024-02-05T05:33:34.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:34.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631494 -lt 442381631493 2024-02-05T05:33:34.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:34.725 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449104 2024-02-05T05:33:34.725 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:34.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:34.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449104 2024-02-05T05:33:34.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:34.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449104 2024-02-05T05:33:34.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449104' 2024-02-05T05:33:34.728 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 335007449104 2024-02-05T05:33:34.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:34.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449104 -lt 335007449104 2024-02-05T05:33:34.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:34.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:34.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:35.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:33:35.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:35.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:35.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:35.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:35.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:35.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:35.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:35.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:33:35.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:35.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:35.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:35.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:33:35.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:35.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:35.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:33:35.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:33:35.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:33:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:33:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:33:36.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:33:36.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:33:36.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:33:36.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:33:36.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:33:36.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:33:38.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:33:38.819 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:33:38.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:33:38.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:33:38.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:33:38.821 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:33:38.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:33:38.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:33:38.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:33:38.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:33:38.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:33:38.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:33:38.847 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:38.846+0000 7f2db7a0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:38.849 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:38.849+0000 7f2db7a0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:38.851 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:38.850+0000 7f2db7a0e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:33:39.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:33:39.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:33:39.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:33:39.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:33:39.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:33:39.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:39.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:33:39.100 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:33:39.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:39.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:39.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:39.378 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:39.377+0000 7f2db7a0e4c0 -1 Falling back to public interface 2024-02-05T05:33:40.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:40.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:40.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:33:40.369 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:40.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:40.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:40.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:41.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:41.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:41.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:33:41.650 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:33:41.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:41.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:41.706 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:33:41.705+0000 7f2db7a0e4c0 -1 osd.0 114 log_to_monitors true 2024-02-05T05:33:41.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:33:42.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:33:42.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:33:42.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:33:42.933 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:33:42.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:33:42.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 119 up_thru 119 down_at 115 last_clean_interval [112,114) [v2:127.0.0.1:6802/1821407100,v1:127.0.0.1:6803/1821407100] [v2:127.0.0.1:6804/1821407100,v1:127.0.0.1:6805/1821407100] exists,up 13a6ee52-551c-4384-a325-a38a39f705c6 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:33:43.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:33:43.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:33:43.314 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:33:43.314 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:33:43.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:33:43.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:33:43.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:33:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:33:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:33:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:33:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:33:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:33:43.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:33:43.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:33:43.758 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:33:43.759 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:33:43.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:33:43.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:43.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:33:43.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108226 2024-02-05T05:33:43.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108226 2024-02-05T05:33:43.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-511101108226' 2024-02-05T05:33:43.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:43.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:33:43.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=442381631497 2024-02-05T05:33:43.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 442381631497 2024-02-05T05:33:43.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-511101108226 1-442381631497' 2024-02-05T05:33:43.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:33:43.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:33:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449107 2024-02-05T05:33:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449107 2024-02-05T05:33:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-511101108226 1-442381631497 2-335007449107' 2024-02-05T05:33:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:44.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-511101108226 2024-02-05T05:33:44.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:44.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:33:44.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-511101108226 2024-02-05T05:33:44.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:44.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108226 2024-02-05T05:33:44.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 511101108226' 2024-02-05T05:33:44.074 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 511101108226 2024-02-05T05:33:44.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:44.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 511101108226 2024-02-05T05:33:44.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:45.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:33:45.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:45.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 511101108226 2024-02-05T05:33:45.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:33:46.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:33:46.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:33:46.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108226 -lt 511101108226 2024-02-05T05:33:46.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:46.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-442381631497 2024-02-05T05:33:46.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:46.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:33:46.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-442381631497 2024-02-05T05:33:46.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:46.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=442381631497 2024-02-05T05:33:46.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 442381631497' 2024-02-05T05:33:46.893 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 442381631497 2024-02-05T05:33:46.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:33:47.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 442381631497 -lt 442381631497 2024-02-05T05:33:47.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:33:47.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-335007449107 2024-02-05T05:33:47.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:33:47.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:33:47.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-335007449107 2024-02-05T05:33:47.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:33:47.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449107 2024-02-05T05:33:47.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 335007449107' 2024-02-05T05:33:47.166 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 335007449107 2024-02-05T05:33:47.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:33:47.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449107 -lt 335007449107 2024-02-05T05:33:47.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:33:47.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:47.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:47.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:33:47.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:33:47.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:33:47.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:33:47.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:33:47.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:33:47.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:33:47.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:33:48.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:33:48.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:33:48.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:33:48.037 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:33:48.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:33:48.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:33:48.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:33:48.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-02-05T05:33:48.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-02-05T05:33:48.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:33:48.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-02-05T05:33:48.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-02-05T05:33:48.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:33:48.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-02-05T05:33:48.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-02-05T05:33:48.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T05:33:48.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T05:33:48.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:48.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:48.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:48.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:33:48.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:48.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T05:33:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-02-05T05:33:49.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:49.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:33:49.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:49.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:33:49.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:49.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:50.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:50.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:50.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:33:50.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:50.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:50.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:50.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:33:50.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:50.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:51.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:51.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:51.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:33:51.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:51.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:51.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:51.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:33:51.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:51.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:33:52.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:52.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:33:53.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:53.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:54.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:54.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:54.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:33:54.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:54.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:33:54.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:54.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:33:54.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:49.281870+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:54.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:33:54.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T05:33:54.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T05:33:54.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-02-05T05:33:54.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T05:33:54.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T05:33:54.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T05:33:54.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T05:33:54.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=119 2024-02-05T05:33:54.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:33:54.599 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:33:54.599 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:33:54.599 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:33:54.599 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:33:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:33:54.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:33:54.622 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:33:54.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:33:54.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:33:54.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:33:54.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-02-05T05:33:54.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-02-05T05:33:54.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-02-05T05:33:54.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T05:33:54.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:33:54.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:54.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:33:54.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:54.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:33:54.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:54.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T05:33:55.206 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T05:31:27.168003+0000 last_deep_scrub_stamp 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:33:55.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:55.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:33:55.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:55.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:33:55.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:55.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:56.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:56.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:56.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:33:56.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:56.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:33:56.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:56.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:33:56.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:56.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:57.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:57.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:57.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:33:57.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:57.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:33:57.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:57.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:33:58.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:58.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:33:59.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:33:59.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:33:59.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:33:59.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:33:59.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:33:59.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:33:59.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:33:59.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:31:27.168003+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:33:59.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:34:00.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:34:00.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:34:00.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:34:00.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:34:00.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:34:00.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:34:00.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:34:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:33:55.458183+0000 '>' 2024-02-05T05:31:27.168003+0000 2024-02-05T05:34:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:34:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T05:34:00.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T05:34:00.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-02-05T05:34:00.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T05:34:00.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T05:34:00.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T05:34:00.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=119 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:34:00.649 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:34:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:34:00.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:34:00.672 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:34:00.672 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:34:00.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:34:00.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:34:00.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:34:00.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-02-05T05:34:00.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-02-05T05:34:00.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-02-05T05:34:01.081 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' removed 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:34:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:34:01.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:34:01.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:34:01.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:34:01.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:34:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:34:01.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:34:01.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:34:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:34:01.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:34:01.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:01.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:34:01.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:34:01.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:34:01.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:34:01.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:01.232 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:01.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:34:01.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:34:01.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:34:01.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:34:01.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:34:01.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:34:01.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:34:01.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:34:01.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:34:01.237 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:34:01.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:34:01.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:34:01.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:34:01.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:01.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:34:01.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:34:01.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:34:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:34:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:34:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:34:01.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:34:01.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:01.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:34:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:34:01.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:34:01.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:34:01.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-02-05T05:34:01.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:34:01.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:34:01.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:34:01.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:01.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:01.613 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:01.613 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:01.614 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:01.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:34:01.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:34:01.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:34:01.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:34:01.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:34:01.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:34:01.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:34:01.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:34:01.806 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:34:01.807 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:34:01.807 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:34:01.807 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:34:01.808 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:01.808 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:01.808 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:34:01.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:34:01.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:34:01.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:34:01.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:34:01.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:34:01.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:34:01.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:34:01.886 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:34:01.886 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:34:01.886 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:34:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:34:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:34:01.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:34:01.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:34:01.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-02-05T05:34:01.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:34:01.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:34:01.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:34:01.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:34:01.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:34:01.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:34:02.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:34:02.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:02.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:02.782 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:02.783 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:02.783 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:02.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:34:02.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:34:02.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:34:02.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-02-05T05:34:02.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:34:02.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-02-05T05:34:02.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:34:02.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:34:02.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:34:02.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8dcd13bb-17e1-4778-a824-1a5028a5e477' 2024-02-05T05:34:02.818 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:34:02.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:34:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKc8BlRMeXMRAAbvLGfuOfXe7kmTM/ejUEJg== 2024-02-05T05:34:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKc8BlRMeXMRAAbvLGfuOfXe7kmTM/ejUEJg=="}' 2024-02-05T05:34:02.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8dcd13bb-17e1-4778-a824-1a5028a5e477 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:34:03.047 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:34:03.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:34:03.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKc8BlRMeXMRAAbvLGfuOfXe7kmTM/ejUEJg== --osd-uuid 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:34:03.085 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:03.084+0000 7ffb2a4654c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:03.097 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:03.096+0000 7ffb2a4654c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:03.099 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:03.097+0000 7ffb2a4654c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:03.099 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:03.098+0000 7ffb2a4654c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:34:03.099 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:03.098+0000 7ffb2a4654c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:34:03.099 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:03.098+0000 7ffb2a4654c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:34:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:03.188+0000 7ffb2a4654c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:34:08.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:34:08.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:34:08.647 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:34:08.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:34:08.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:34:08.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:34:08.999 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:34:09.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:34:09.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:34:09.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:34:09.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:34:09.025 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:09.024+0000 7f96380a74c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:09.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:09.026+0000 7f96380a74c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:09.028 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:09.027+0000 7f96380a74c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:09.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:34:09.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:34:09.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:34:09.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:34:09.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:34:09.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:09.271 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:34:09.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:34:09.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:09.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:09.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:09.561 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:09.560+0000 7f96380a74c0 -1 Falling back to public interface 2024-02-05T05:34:10.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:10.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:10.537 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:34:10.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:34:10.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:10.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:10.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:11.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:11.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:11.813 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:34:11.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:34:11.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:11.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:11.920 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:11.919+0000 7f96380a74c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:34:12.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:13.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:13.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:13.105 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:34:13.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:34:13.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:13.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:13.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:14.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:14.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:14.382 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:34:14.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:34:14.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:14.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:14.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:15.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:15.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:15.707 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:34:15.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:34:15.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:15.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:15.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:16.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:16.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:16.988 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:34:16.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:34:16.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:16.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:17.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:18.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:18.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:18.271 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:34:18.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:34:18.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:18.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:18.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:19.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:19.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:19.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:34:19.552 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:34:19.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:19.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:19.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:20.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:20.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:20.876 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:34:20.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:34:20.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:20.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:21.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:22.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:22.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:22.154 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:34:22.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:34:22.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:22.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:22.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:23.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:23.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:23.513 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:34:23.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:34:23.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:23.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:24.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:25.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:25.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:25.163 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:34:25.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:34:25.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:25.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:25.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:26.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:26.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:26.652 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:34:26.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:34:26.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:26.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:27.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:28.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:28.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:28.273 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:34:28.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:34:28.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:28.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:28.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:29.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:29.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:29.862 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:34:29.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:34:29.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:29.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:30.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:31.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:31.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:31.395 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:34:31.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:34:31.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:31.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:31.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:32.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:32.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:32.674 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:34:32.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:34:32.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:32.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:33.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:34.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:34.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:34.131 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:34:34.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:34:34.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:34.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:34:34.493 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2951245147,v1:127.0.0.1:6803/2951245147] [v2:127.0.0.1:6804/2951245147,v1:127.0.0.1:6805/2951245147] exists,up 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:34:34.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:34:34.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:34:34.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:34:34.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:34:34.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-02-05T05:34:34.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:34:34.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:34:34.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:34:34.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 92637a52-3bf7-4a26-9dfd-8decec4c2bed' 2024-02-05T05:34:34.500 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:34:34.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:34:34.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqc8BlNuefHhAACc/YryNxrTTuexmy0jFXKQ== 2024-02-05T05:34:34.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqc8BlNuefHhAACc/YryNxrTTuexmy0jFXKQ=="}' 2024-02-05T05:34:34.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 92637a52-3bf7-4a26-9dfd-8decec4c2bed -i td/osd-scrub-repair/1/new.json 2024-02-05T05:34:34.942 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:34:34.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:34:34.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqc8BlNuefHhAACc/YryNxrTTuexmy0jFXKQ== --osd-uuid 92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:34:34.983 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:34.981+0000 7f6511a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:35.026+0000 7f6511a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:35.028 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:35.027+0000 7f6511a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:35.029 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:35.027+0000 7f6511a144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:34:35.029 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:35.027+0000 7f6511a144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:34:35.029 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:35.027+0000 7f6511a144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:34:35.094 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:35.093+0000 7f6511a144c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:34:41.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:34:41.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:34:41.034 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:34:41.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:34:41.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:34:41.377 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:34:41.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:34:41.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:34:41.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:34:41.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:34:41.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:34:41.403 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:41.402+0000 7f0ec9c144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:41.405 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:41.404+0000 7f0ec9c144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:41.406 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:41.405+0000 7f0ec9c144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:34:41.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:34:41.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:34:41.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:34:41.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:34:41.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:34:41.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:41.654 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:34:41.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:34:41.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:41.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:41.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:41.938 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:41.937+0000 7f0ec9c144c0 -1 Falling back to public interface 2024-02-05T05:34:42.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:42.920 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:34:42.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:42.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:34:42.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:42.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:44.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:44.198 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:34:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:34:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:44.276 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:34:44.275+0000 7f0ec9c144c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:34:44.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:45.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:45.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:45.486 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:34:45.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:34:45.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:45.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:45.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:46.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:46.763 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:34:46.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:46.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:34:46.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:46.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:47.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:48.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:48.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:48.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:34:48.148 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:34:48.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:48.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:48.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:49.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:49.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:49.427 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:34:49.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:34:49.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:49.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:49.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:50.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:50.703 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:34:50.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:50.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:34:50.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:50.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:50.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:51.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:51.994 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:34:51.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:51.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:34:51.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:51.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:52.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:53.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:53.357 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:34:53.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:53.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:34:53.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:53.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:53.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:54.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:54.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:54.635 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:34:54.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:34:54.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:54.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:54.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:55.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:55.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:55.992 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:34:55.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:34:55.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:55.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:56.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:57.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:57.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:57.275 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:34:57.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:34:57.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:57.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:58.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:34:59.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:34:59.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:34:59.287 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:34:59.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:34:59.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:34:59.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:34:59.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:00.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:00.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:00.655 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:35:00.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:35:00.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:00.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:01.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:02.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:02.193 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:35:02.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:02.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:35:02.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:02.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:03.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:04.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:04.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:04.080 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:35:04.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:35:04.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:04.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:04.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:05.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:05.652 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:35:05.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:05.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:35:05.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:05.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:06.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:07.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:07.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:07.049 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:35:07.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:35:07.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:07.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:07.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:08.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:08.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:08.455 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:35:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:35:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:35:08.726 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3862489810,v1:127.0.0.1:6811/3862489810] [v2:127.0.0.1:6812/3862489810,v1:127.0.0.1:6813/3862489810] exists,up 92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:35:08.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:35:08.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:35:08.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:35:08.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:35:08.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:35:08.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:35:08.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:35:08.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:35:08.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:35:08.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:35:08.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:35:08.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:35:08.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:35:08.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:35:08.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:35:08.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:35:08.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:35:08.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:35:08.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:35:08.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:35:08.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26c671c4-a676-40a7-bfc5-f021767b221d 2024-02-05T05:35:08.738 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 26c671c4-a676-40a7-bfc5-f021767b221d 2024-02-05T05:35:08.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 26c671c4-a676-40a7-bfc5-f021767b221d' 2024-02-05T05:35:08.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:35:08.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMc8BlHTGFLBAA1k4SEiGJp2YR365XkgvEdA== 2024-02-05T05:35:08.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMc8BlHTGFLBAA1k4SEiGJp2YR365XkgvEdA=="}' 2024-02-05T05:35:08.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26c671c4-a676-40a7-bfc5-f021767b221d -i td/osd-scrub-repair/2/new.json 2024-02-05T05:35:09.103 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:35:09.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:35:09.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMc8BlHTGFLBAA1k4SEiGJp2YR365XkgvEdA== --osd-uuid 26c671c4-a676-40a7-bfc5-f021767b221d 2024-02-05T05:35:09.143 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:09.142+0000 7f40ac4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:09.180 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:09.178+0000 7f40ac4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:09.180 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:09.179+0000 7f40ac4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:09.181 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:09.180+0000 7f40ac4144c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:35:09.181 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:09.180+0000 7f40ac4144c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:35:09.181 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:09.180+0000 7f40ac4144c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:35:09.255 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:09.254+0000 7f40ac4144c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:35:15.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:35:15.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:35:15.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:35:15.205 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:35:15.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:35:15.553 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:35:15.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:35:15.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:35:15.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:35:15.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:35:15.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:35:15.579 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:15.578+0000 7f61343bc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:15.581 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:15.580+0000 7f61343bc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:15.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:15.581+0000 7f61343bc4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:35:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:35:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:35:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:35:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:35:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:35:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:35:15.823 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:35:15.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:15.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:16.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:16.111 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:16.110+0000 7f61343bc4c0 -1 Falling back to public interface 2024-02-05T05:35:17.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:17.089 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:35:17.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:17.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:35:17.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:17.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:17.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:18.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:18.361 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:35:18.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:18.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:35:18.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:18.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:18.470 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:35:18.469+0000 7f61343bc4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:35:18.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:19.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:19.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:19.681 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:35:19.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:35:19.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:19.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:19.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:20.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:20.959 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:35:20.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:20.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:35:20.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:20.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:21.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:35:22.435 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:35:22.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:22.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:22.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:23.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:23.717 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:35:23.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:23.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:35:23.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:23.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:23.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:24.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:24.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:24.994 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:35:24.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:35:24.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:24.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:25.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:26.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:26.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:26.477 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:35:26.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:35:26.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:26.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:26.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:27.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:35:27.819 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:35:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:27.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:28.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:29.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:29.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:29.095 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:35:29.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:35:29.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:29.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:29.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:30.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:30.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:30.425 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:35:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:35:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:31.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:32.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:32.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:32.035 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:35:32.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:35:32.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:32.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:32.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:33.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:33.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:33.701 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:35:33.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:35:33.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:33.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:34.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:35.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:35.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:35.067 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:35:35.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:35:35.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:35.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:35.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:36.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:36.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:36.628 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:35:36.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:35:36.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:36.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:37.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:38.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:38.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:38.282 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:35:38.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:35:38.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:38.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:39.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:40.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:40.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:40.011 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:35:40.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:35:40.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:40.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:40.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:35:41.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:35:41.361 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:35:41.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:35:41.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:35:41.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:35:41.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:35:41.688 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3904429296,v1:127.0.0.1:6819/3904429296] [v2:127.0.0.1:6820/3904429296,v1:127.0.0.1:6821/3904429296] exists,up 26c671c4-a676-40a7-bfc5-f021767b221d 2024-02-05T05:35:41.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:35:41.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:35:41.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:35:41.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-02-05T05:35:41.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:35:41.952 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:35:41.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:35:41.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:35:42.332 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:35:42.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:35:43.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:35:45.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-02-05T05:35:45.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T05:35:46.164 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'foo' created 2024-02-05T05:35:46.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:35:47.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-02-05T05:35:47.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:35:47.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:35:47.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:35:47.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:35:47.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-02-05T05:35:47.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:35:47.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:35:48.226 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:35:48.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:35:49.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:35:49.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:35:49.677 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 3 allow_ec_overwrites to true 2024-02-05T05:35:49.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:35:49.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:35:49.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:35:49.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:35:49.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:35:49.690 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:35:49.690 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:35:49.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:35:49.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:35:49.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:35:49.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:35:49.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:35:49.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:35:49.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:35:49.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:35:49.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:35:50.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:35:50.100 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:35:50.101 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:35:50.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:35:50.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:35:50.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:35:50.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T05:35:50.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T05:35:50.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T05:35:50.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:35:50.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:35:50.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T05:35:50.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T05:35:50.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672972' 2024-02-05T05:35:50.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:35:50.329 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:35:50.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-05T05:35:50.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-05T05:35:50.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672972 2-64424509447' 2024-02-05T05:35:50.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:35:50.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T05:35:50.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:35:50.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:35:50.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T05:35:50.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:35:50.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T05:35:50.434 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836500 2024-02-05T05:35:50.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T05:35:50.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:35:50.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-02-05T05:35:50.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:35:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:35:51.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:35:51.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836500 2024-02-05T05:35:51.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:35:51.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T05:35:51.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:35:51.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:35:51.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T05:35:51.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:35:51.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T05:35:51.978 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672972 2024-02-05T05:35:51.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T05:35:51.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:35:52.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T05:35:52.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:35:52.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-05T05:35:52.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:35:52.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:35:52.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-05T05:35:52.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:35:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-05T05:35:52.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-05T05:35:52.249 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509447 2024-02-05T05:35:52.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:35:52.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-05T05:35:52.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:35:52.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:35:52.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:35:52.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:35:52.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:35:52.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:35:52.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:35:52.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:35:52.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:35:52.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:35:52.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:35:53.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:35:53.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:35:53.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:35:53.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:35:53.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:35:53.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:35:53.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:35:53.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:35:53.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:35:53.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:35:53.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:35:53.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:35:53.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:35:53.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:35:53.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:35:53.840 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:35:53.840 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:35:53.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:35:53.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:35:53.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:35:53.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-02-05T05:35:53.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-02-05T05:35:53.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-02-05T05:35:53.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:35:53.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:35:54.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-05T05:35:54.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-05T05:35:54.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672974' 2024-02-05T05:35:54.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:35:54.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:35:54.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T05:35:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T05:35:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672974 2-64424509449' 2024-02-05T05:35:54.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:35:54.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-02-05T05:35:54.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:35:54.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:35:54.152 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-02-05T05:35:54.152 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:35:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-02-05T05:35:54.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-02-05T05:35:54.153 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836502 2024-02-05T05:35:54.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:35:54.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-02-05T05:35:54.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:35:55.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:35:55.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:35:55.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-02-05T05:35:55.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:35:55.703 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-05T05:35:55.703 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:35:55.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:35:55.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-05T05:35:55.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:35:55.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-05T05:35:55.706 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672974 2024-02-05T05:35:55.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-05T05:35:55.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:35:55.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-02-05T05:35:55.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:35:55.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T05:35:55.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:35:55.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:35:55.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T05:35:55.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:35:55.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T05:35:55.976 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509449 2024-02-05T05:35:55.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T05:35:55.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:35:56.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T05:35:56.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:35:56.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:35:56.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:35:56.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:35:56.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:35:56.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:35:56.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:35:56.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:35:56.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:35:56.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:35:56.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:35:56.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:35:56.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:35:56.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:35:56.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:35:57.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:35:57.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:35:57.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:35:57.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:35:57.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:35:57.680 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:35:57.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:35:58.057 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:35:58.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:35:58.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:35:58.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:35:58.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-02-05T05:35:58.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:35:58.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:35:58.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-02-05T05:35:58.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-02-05T05:35:58.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:35:58.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:35:58.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:35:59.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:35:59.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:35:59.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:35:59.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:35:59.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:35:59.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:35:59.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:35:59.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:36:01.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:36:01.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:36:01.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:36:01.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:36:01.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:36:01.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:36:01.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:36:01.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:36:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:36:01.270 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:01.269+0000 7f6be0a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:01.272 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:01.271+0000 7f6be0a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:01.273 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:01.273+0000 7f6be0a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:36:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:36:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:36:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:36:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:36:01.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:01.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:36:01.520 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:36:01.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:01.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:01.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:01.795 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:01.794+0000 7f6be0a144c0 -1 Falling back to public interface 2024-02-05T05:36:02.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:02.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:02.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:36:02.789 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:02.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:02.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:03.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:04.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:04.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:04.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:36:04.067 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:36:04.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:04.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:04.129 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:04.127+0000 7f6be0a144c0 -1 osd.1 36 log_to_monitors true 2024-02-05T05:36:04.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:05.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:05.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:05.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:36:05.412 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:36:05.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:05.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/1313971559,v1:127.0.0.1:6811/1313971559] [v2:127.0.0.1:6812/1313971559,v1:127.0.0.1:6813/1313971559] exists,up 92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:36:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:36:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:36:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:36:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:36:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:36:05.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:36:05.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:36:05.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:36:05.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:36:05.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:36:05.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:36:05.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:36:06.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:36:06.063 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:06.063 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:36:06.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:36:06.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:06.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:36:06.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-02-05T05:36:06.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-02-05T05:36:06.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-02-05T05:36:06.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:06.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:36:06.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-02-05T05:36:06.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-02-05T05:36:06.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-171798691842' 2024-02-05T05:36:06.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:06.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:36:06.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T05:36:06.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T05:36:06.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-171798691842 2-64424509452' 2024-02-05T05:36:06.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:06.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-02-05T05:36:06.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:06.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:36:06.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-02-05T05:36:06.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:06.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-02-05T05:36:06.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-02-05T05:36:06.436 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836506 2024-02-05T05:36:06.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:06.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836506 2024-02-05T05:36:06.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:36:07.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:36:07.703 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:07.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-02-05T05:36:07.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:07.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-02-05T05:36:07.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:07.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:36:07.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-02-05T05:36:07.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:07.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-02-05T05:36:07.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-02-05T05:36:07.982 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 171798691842 2024-02-05T05:36:07.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:36:08.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-02-05T05:36:08.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:08.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T05:36:08.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:08.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:36:08.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T05:36:08.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:08.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T05:36:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T05:36:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509452 2024-02-05T05:36:08.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:36:08.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T05:36:08.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:36:08.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:08.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:08.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:36:08.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:36:08.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:36:08.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:36:08.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:36:08.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:36:08.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:36:08.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:36:09.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:36:09.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:36:09.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:09.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:09.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:36:09.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:36:09.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:36:09.849 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:36:09.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:36:10.309 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:36:10.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:36:10.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:36:10.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:36:10.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-02-05T05:36:10.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:36:10.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:36:10.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-02-05T05:36:10.516 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T05:36:10.516 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T05:36:10.516 INFO:tasks.workunit.client.0.smithi188.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.1467e-05 s, 33.3 MB/s 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:36:10.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:36:10.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:36:10.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:36:11.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:36:13.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:36:13.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:36:13.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:36:13.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:36:13.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:36:13.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:36:13.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:36:13.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:36:13.589 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:36:13.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:36:13.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:36:13.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:36:13.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:36:13.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:36:13.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:36:13.616 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:13.615+0000 7f6845a104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:13.618 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:13.617+0000 7f6845a104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:13.619 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:13.618+0000 7f6845a104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:13.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:36:13.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:36:13.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:36:13.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:36:13.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:36:13.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:13.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:36:13.866 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:36:13.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:13.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:14.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:14.138 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:14.138+0000 7f6845a104c0 -1 Falling back to public interface 2024-02-05T05:36:15.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:15.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:15.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:36:15.133 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:15.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:15.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:15.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:16.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:16.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:16.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:36:16.411 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:36:16.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:16.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:16.506 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:16.505+0000 7f6845a104c0 -1 osd.0 44 log_to_monitors true 2024-02-05T05:36:16.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:36:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:36:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:17.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:18.098 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/1813645454,v1:127.0.0.1:6803/1813645454] [v2:127.0.0.1:6804/1813645454,v1:127.0.0.1:6805/1813645454] exists,up 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:36:18.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:36:18.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:36:18.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:36:18.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:36:18.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:36:18.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:36:18.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:36:18.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:36:18.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:36:18.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:36:18.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:36:18.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:36:18.574 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:18.574 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:36:18.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:36:18.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:18.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:36:18.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-02-05T05:36:18.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-02-05T05:36:18.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506' 2024-02-05T05:36:18.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:18.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:36:18.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-02-05T05:36:18.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-02-05T05:36:18.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845' 2024-02-05T05:36:18.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:18.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:36:18.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-02-05T05:36:18.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-02-05T05:36:18.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845 2-64424509456' 2024-02-05T05:36:18.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:18.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397506 2024-02-05T05:36:18.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:18.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:36:18.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397506 2024-02-05T05:36:18.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:18.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-02-05T05:36:18.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-02-05T05:36:18.887 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 210453397506 2024-02-05T05:36:18.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:19.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-02-05T05:36:19.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:19.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-02-05T05:36:19.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:19.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:36:19.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-02-05T05:36:19.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:19.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-02-05T05:36:19.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-02-05T05:36:19.158 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 171798691845 2024-02-05T05:36:19.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:36:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-02-05T05:36:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:36:20.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:36:20.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:36:20.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691845 2024-02-05T05:36:20.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:36:21.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:36:21.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:36:21.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691845 2024-02-05T05:36:21.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:21.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-02-05T05:36:21.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:21.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:36:21.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-02-05T05:36:21.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:21.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-02-05T05:36:21.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-02-05T05:36:21.982 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509456 2024-02-05T05:36:21.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:36:22.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-02-05T05:36:22.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:36:22.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:22.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:36:22.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:36:22.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:36:22.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:36:22.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:22.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:36:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:36:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:36:23.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:36:23.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:36:23.552 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:36:23.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:36:23.929 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:36:23.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:36:23.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:36:23.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:36:24.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-02-05T05:36:24.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:36:24.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:36:24.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:36:24.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:36:24.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:36:24.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:36:24.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:36:24.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:36:24.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:36:24.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:36:24.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:36:24.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:36:24.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:36:24.272 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:36:24.272 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:36:24.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:36:24.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:36:24.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:36:24.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:36:24.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-02-05T05:36:24.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:36:24.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:36:24.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:36:24.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:36:24.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:36:24.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-02-05T05:36:26.253 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:36:26.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:26.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:36:26.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:36:26.774 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:36:26.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:36:26.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:36:26.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:36:26.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:36:26.799 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:26.798+0000 7f4840c1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:26.801 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:26.800+0000 7f4840c1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:26.802 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:26.801+0000 7f4840c1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:27.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:36:27.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:36:27.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:36:27.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:36:27.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:36:27.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:27.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:36:27.052 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:36:27.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:27.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:27.317 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:27.316+0000 7f4840c1a4c0 -1 Falling back to public interface 2024-02-05T05:36:27.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:28.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:28.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:28.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:36:28.359 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:28.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:28.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:28.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:29.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:29.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:29.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:36:29.633 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:36:29.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:29.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:29.687 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:29.686+0000 7f4840c1a4c0 -1 osd.1 53 log_to_monitors true 2024-02-05T05:36:29.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:30.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:30.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:30.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:36:30.917 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:36:30.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:30.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:36:31.229 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/175916573,v1:127.0.0.1:6811/175916573] [v2:127.0.0.1:6812/175916573,v1:127.0.0.1:6813/175916573] exists,up 92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:36:31.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:36:31.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:36:31.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:36:31.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:36:31.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:36:31.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:36:31.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:36:31.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:36:31.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:36:31.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:36:31.673 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:31.673 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:36:31.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:36:31.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:31.673 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:36:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397510 2024-02-05T05:36:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397510 2024-02-05T05:36:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397510' 2024-02-05T05:36:31.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:31.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:36:31.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-02-05T05:36:31.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-02-05T05:36:31.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397510 1-249108103170' 2024-02-05T05:36:31.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:31.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:36:31.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-02-05T05:36:31.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-02-05T05:36:31.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397510 1-249108103170 2-64424509460' 2024-02-05T05:36:31.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:31.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397510 2024-02-05T05:36:31.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:31.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:36:31.984 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397510 2024-02-05T05:36:31.984 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:31.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397510 2024-02-05T05:36:31.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397510' 2024-02-05T05:36:31.985 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 210453397510 2024-02-05T05:36:31.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:32.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397508 -lt 210453397510 2024-02-05T05:36:32.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:36:33.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:36:33.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:33.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397510 2024-02-05T05:36:33.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:33.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-02-05T05:36:33.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:33.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:36:33.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-02-05T05:36:33.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:33.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-02-05T05:36:33.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-02-05T05:36:33.531 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 249108103170 2024-02-05T05:36:33.531 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:36:33.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-02-05T05:36:33.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:33.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-02-05T05:36:33.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:33.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:36:33.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-02-05T05:36:33.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:33.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-02-05T05:36:33.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-02-05T05:36:33.804 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509460 2024-02-05T05:36:33.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:36:34.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-02-05T05:36:34.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:36:34.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:34.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:36:34.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:36:34.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:36:34.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:36:34.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:34.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:35.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:36:35.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:36:35.428 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:36:35.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:36:35.955 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:36:35.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:36:35.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:36:35.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:36:36.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-02-05T05:36:36.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:36:36.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:36:36.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-02-05T05:36:36.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:36:36.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:36:36.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:36:36.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-02-05T05:36:38.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:36:38.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:36:38.857 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:36:38.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:36:38.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:36:38.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:36:38.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:36:38.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:36:38.859 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:36:38.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:36:38.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:36:38.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:36:38.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:36:38.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:36:38.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:36:38.886 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:38.885+0000 7f854ba254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:38.888 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:38.887+0000 7f854ba254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:38.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:38.889+0000 7f854ba254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:39.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:39.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:39.404+0000 7f854ba254c0 -1 Falling back to public interface 2024-02-05T05:36:40.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:40.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:40.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:36:40.405 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:40.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:40.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:40.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:41.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:41.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:41.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:36:41.676 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:36:41.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:41.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:41.708 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:41.707+0000 7f854ba254c0 -1 osd.0 62 log_to_monitors true 2024-02-05T05:36:42.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:43.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:43.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:43.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:36:43.076 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:36:43.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:43.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:36:43.595 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 67 up_thru 67 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/393723409,v1:127.0.0.1:6803/393723409] [v2:127.0.0.1:6804/393723409,v1:127.0.0.1:6805/393723409] exists,up 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:36:43.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:36:43.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:36:43.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:36:43.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:36:43.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:36:43.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:36:43.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:36:43.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:36:43.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:36:43.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:36:43.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:36:43.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:36:43.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:36:43.946 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:43.946 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:36:43.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:36:43.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:43.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:36:44.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808834 2024-02-05T05:36:44.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808834 2024-02-05T05:36:44.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834' 2024-02-05T05:36:44.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:44.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:36:44.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103173 2024-02-05T05:36:44.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103173 2024-02-05T05:36:44.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103173' 2024-02-05T05:36:44.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:44.152 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:36:44.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-02-05T05:36:44.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-02-05T05:36:44.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808834 1-249108103173 2-64424509463' 2024-02-05T05:36:44.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:44.254 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808834 2024-02-05T05:36:44.254 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:44.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:36:44.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808834 2024-02-05T05:36:44.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:44.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808834 2024-02-05T05:36:44.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808834' 2024-02-05T05:36:44.257 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 287762808834 2024-02-05T05:36:44.257 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:44.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 287762808834 2024-02-05T05:36:44.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:36:45.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:36:45.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:45.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808834 -lt 287762808834 2024-02-05T05:36:45.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:45.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103173 2024-02-05T05:36:45.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:45.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:36:45.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103173 2024-02-05T05:36:45.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:45.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103173 2024-02-05T05:36:45.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-02-05T05:36:45.802 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 249108103173 2024-02-05T05:36:45.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:36:46.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103173 -lt 249108103173 2024-02-05T05:36:46.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:46.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-02-05T05:36:46.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:46.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:36:46.071 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-02-05T05:36:46.071 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:46.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-02-05T05:36:46.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-02-05T05:36:46.075 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 64424509463 2024-02-05T05:36:46.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:36:46.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-02-05T05:36:46.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:36:46.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:46.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:46.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:36:46.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:36:46.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:36:46.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:36:46.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:36:46.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:36:46.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:36:46.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:36:46.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:36:46.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:36:46.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:46.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:47.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:36:47.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:36:47.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:36:47.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:36:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:36:47.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:36:47.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:36:47.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:36:47.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:36:47.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:36:47.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:36:47.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:36:47.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:36:47.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:36:47.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-02-05T05:36:49.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:36:49.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:36:49.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:36:49.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:36:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:36:49.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:36:49.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:36:49.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:36:49.505 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.2 2024-02-05T05:36:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:36:49.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:36:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:36:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:36:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:36:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:36:49.532 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:49.531+0000 7f7b220494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:49.534 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:49.533+0000 7f7b220494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:49.534+0000 7f7b220494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:49.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:36:49.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:49.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:50.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:50.061 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:50.060+0000 7f7b220494c0 -1 Falling back to public interface 2024-02-05T05:36:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:36:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:51.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:52.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:52.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:52.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:36:52.321 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:36:52.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:52.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:52.396 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:52.395+0000 7f7b220494c0 -1 osd.2 69 log_to_monitors true 2024-02-05T05:36:52.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:53.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:36:53.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:53.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:36:53.601 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:36:53.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:53.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:53.916 INFO:tasks.workunit.client.0.smithi188.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/1093059332,v1:127.0.0.1:6819/1093059332] [v2:127.0.0.1:6820/1093059332,v1:127.0.0.1:6821/1093059332] exists,up 26c671c4-a676-40a7-bfc5-f021767b221d 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:36:53.917 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:36:53.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:36:53.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:36:53.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:36:54.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:36:54.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:36:54.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:36:54.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:36:54.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:36:54.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:36:54.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:36:54.296 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:36:54.296 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:36:54.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:36:54.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:54.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:36:54.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808837 2024-02-05T05:36:54.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808837 2024-02-05T05:36:54.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837' 2024-02-05T05:36:54.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:54.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:36:54.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103176 2024-02-05T05:36:54.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103176 2024-02-05T05:36:54.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103176' 2024-02-05T05:36:54.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:36:54.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:36:54.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-02-05T05:36:54.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-02-05T05:36:54.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808837 1-249108103176 2-313532612610' 2024-02-05T05:36:54.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:54.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808837 2024-02-05T05:36:54.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:54.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:36:54.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808837 2024-02-05T05:36:54.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:54.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808837 2024-02-05T05:36:54.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808837' 2024-02-05T05:36:54.610 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 287762808837 2024-02-05T05:36:54.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:36:54.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808837 -lt 287762808837 2024-02-05T05:36:54.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:54.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103176 2024-02-05T05:36:54.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:54.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:36:54.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103176 2024-02-05T05:36:54.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:54.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103176 2024-02-05T05:36:54.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103176' 2024-02-05T05:36:54.879 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 249108103176 2024-02-05T05:36:54.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:36:55.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103176 -lt 249108103176 2024-02-05T05:36:55.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:36:55.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-02-05T05:36:55.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:36:55.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:36:55.152 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-02-05T05:36:55.152 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:36:55.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-02-05T05:36:55.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-02-05T05:36:55.153 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 313532612610 2024-02-05T05:36:55.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:36:55.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-02-05T05:36:55.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:36:55.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:55.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:55.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:36:55.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:36:55.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:36:55.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:36:55.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:36:55.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:36:55.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:36:55.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:36:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:36:56.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:36:56.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:36:56.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:36:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:36:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:36:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:36:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-02-05T05:36:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:36:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:36:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:36:56.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:36:56.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:36:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:36:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:36:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:36:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T05:36:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:36:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:36:56.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-02-05T05:36:58.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:36:58.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:36:58.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:36:58.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:36:58.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:36:58.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:36:58.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:36:58.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:36:58.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.2 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:36:58.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:36:58.785 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:58.784+0000 7f49a84144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:58.787 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:58.787+0000 7f49a84144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:58.789 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:58.788+0000 7f49a84144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:36:59.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:36:59.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:36:59.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:36:59.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:36:59.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:36:59.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:36:59.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:36:59.036 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:36:59.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:36:59.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:36:59.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:36:59.305 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:36:59.305+0000 7f49a84144c0 -1 Falling back to public interface 2024-02-05T05:37:00.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:00.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:00.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:00.302 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:00.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:00.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:37:00.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:01.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:01.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:01.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:01.576 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:37:01.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:01.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:37:01.616 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:01.613+0000 7f49a84144c0 -1 osd.2 75 log_to_monitors true 2024-02-05T05:37:01.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:02.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:02.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:02.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:37:02.859 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:37:02.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:02.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:37:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:osd.2 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/2302119171,v1:127.0.0.1:6819/2302119171] [v2:127.0.0.1:6820/2302119171,v1:127.0.0.1:6821/2302119171] exists,up 26c671c4-a676-40a7-bfc5-f021767b221d 2024-02-05T05:37:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:37:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:37:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:37:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:37:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:37:03.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:37:03.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:37:03.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:37:03.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:37:03.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:37:03.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:37:03.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:37:03.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:37:03.766 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:03.766 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:37:03.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:37:03.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:03.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:37:03.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808840 2024-02-05T05:37:03.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808840 2024-02-05T05:37:03.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840' 2024-02-05T05:37:03.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:03.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:37:04.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103179 2024-02-05T05:37:04.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103179 2024-02-05T05:37:04.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103179' 2024-02-05T05:37:04.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:04.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:37:04.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-02-05T05:37:04.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-02-05T05:37:04.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808840 1-249108103179 2-343597383682' 2024-02-05T05:37:04.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:04.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808840 2024-02-05T05:37:04.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:04.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:37:04.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808840 2024-02-05T05:37:04.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:04.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808840 2024-02-05T05:37:04.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808840' 2024-02-05T05:37:04.118 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 287762808840 2024-02-05T05:37:04.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:04.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808839 -lt 287762808840 2024-02-05T05:37:04.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:37:05.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:37:05.388 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:05.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808840 -lt 287762808840 2024-02-05T05:37:05.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:05.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103179 2024-02-05T05:37:05.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:05.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:37:05.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103179 2024-02-05T05:37:05.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:05.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103179 2024-02-05T05:37:05.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103179' 2024-02-05T05:37:05.659 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 249108103179 2024-02-05T05:37:05.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:37:05.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103179 -lt 249108103179 2024-02-05T05:37:05.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:05.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-02-05T05:37:05.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:05.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:37:05.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-02-05T05:37:05.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:05.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-02-05T05:37:05.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-02-05T05:37:05.930 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 343597383682 2024-02-05T05:37:05.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:37:06.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-02-05T05:37:06.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:37:06.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:06.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:37:06.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:37:06.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:37:06.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:37:06.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:06.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:07.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:37:07.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:37:07.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:37:07.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-02-05T05:37:07.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:37:07.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:37:07.515 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:37:07.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:37:08.092 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:37:08.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:37:08.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:37:08.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:37:08.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-02-05T05:37:08.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:37:08.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:37:08.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-02-05T05:37:08.250 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records in 2024-02-05T05:37:08.250 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records out 2024-02-05T05:37:08.250 INFO:tasks.workunit.client.0.smithi188.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.0586e-05 s, 58.0 MB/s 2024-02-05T05:37:08.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:37:08.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:37:08.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:37:08.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:37:08.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:37:08.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:37:08.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:37:08.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:37:08.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:37:08.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:37:08.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:37:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:37:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:37:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:37:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:37:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:37:10.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:37:10.561 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:37:10.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:37:10.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:37:10.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:37:10.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:37:10.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:37:10.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:37:10.588 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:10.587+0000 7f85d5a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:10.589+0000 7f85d5a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:10.591 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:10.591+0000 7f85d5a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:37:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:37:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:37:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:37:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:37:10.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:10.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:37:10.841 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:37:10.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:10.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:11.106 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:11.105+0000 7f85d5a144c0 -1 Falling back to public interface 2024-02-05T05:37:11.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:12.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:12.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:12.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:12.122 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:12.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:12.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:12.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:13.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:13.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:13.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:13.400 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:37:13.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:13.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:13.581 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:13.580+0000 7f85d5a144c0 -1 osd.1 85 log_to_monitors true 2024-02-05T05:37:13.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:14.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:14.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:14.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:37:14.740 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:37:14.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:14.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 86 last_clean_interval [58,85) [v2:127.0.0.1:6810/2832810425,v1:127.0.0.1:6811/2832810425] [v2:127.0.0.1:6812/2832810425,v1:127.0.0.1:6813/2832810425] exists,up 92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:37:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:37:15.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:37:15.173 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:37:15.173 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:37:15.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:37:15.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:37:15.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:37:15.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:37:15.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:37:15.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:37:15.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:37:15.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:37:15.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:37:15.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:37:15.568 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:15.568 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:37:15.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:37:15.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:15.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:37:15.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=287762808843 2024-02-05T05:37:15.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 287762808843 2024-02-05T05:37:15.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808843' 2024-02-05T05:37:15.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:15.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:37:15.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-02-05T05:37:15.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-02-05T05:37:15.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808843 1-382252089346' 2024-02-05T05:37:15.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:15.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:37:15.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383685 2024-02-05T05:37:15.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383685 2024-02-05T05:37:15.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-287762808843 1-382252089346 2-343597383685' 2024-02-05T05:37:15.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:15.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-287762808843 2024-02-05T05:37:15.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:15.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:37:15.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-287762808843 2024-02-05T05:37:15.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:15.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=287762808843 2024-02-05T05:37:15.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 287762808843' 2024-02-05T05:37:15.917 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 287762808843 2024-02-05T05:37:15.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:16.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808842 -lt 287762808843 2024-02-05T05:37:16.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:37:17.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:37:17.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:17.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 287762808843 -lt 287762808843 2024-02-05T05:37:17.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:17.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-02-05T05:37:17.455 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:17.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:37:17.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-02-05T05:37:17.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:17.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-02-05T05:37:17.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-02-05T05:37:17.458 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 382252089346 2024-02-05T05:37:17.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:37:17.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-02-05T05:37:17.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:17.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383685 2024-02-05T05:37:17.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:17.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:37:17.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383685 2024-02-05T05:37:17.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:17.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383685 2024-02-05T05:37:17.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383685' 2024-02-05T05:37:17.729 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 343597383685 2024-02-05T05:37:17.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:37:17.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383685 -lt 343597383685 2024-02-05T05:37:17.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:37:17.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:17.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:37:18.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:37:18.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:37:18.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:37:18.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:18.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:18.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:37:18.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:37:18.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:37:18.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:37:18.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:37:19.319 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:37:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:37:19.705 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:37:19.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:37:19.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:37:19.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:37:19.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:37:19.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:37:19.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:37:19.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:37:19.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:37:19.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:37:19.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:37:21.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:37:21.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:37:21.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:37:21.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:37:21.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:37:21.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:37:21.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:37:21.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:37:22.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:22.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:37:22.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:37:22.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:37:22.817 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:37:22.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:37:22.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:37:22.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:37:22.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:37:22.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:37:22.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:37:22.844 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:22.843+0000 7f62160164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:22.846 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:22.845+0000 7f62160164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:22.847 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:22.847+0000 7f62160164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:23.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:37:23.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:37:23.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:37:23.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:37:23.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:37:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:37:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:37:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:23.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:23.366 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:23.365+0000 7f62160164c0 -1 Falling back to public interface 2024-02-05T05:37:23.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:24.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:24.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:24.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:24.389 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:24.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:24.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:24.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:25.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:25.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:25.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:25.657 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:37:25.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:25.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:25.734 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:25.733+0000 7f62160164c0 -1 osd.0 93 log_to_monitors true 2024-02-05T05:37:26.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:27.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:27.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:27.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:37:27.015 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:37:27.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:27.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [67,93) [v2:127.0.0.1:6802/3511554635,v1:127.0.0.1:6803/3511554635] [v2:127.0.0.1:6804/3511554635,v1:127.0.0.1:6805/3511554635] exists,up 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:37:27.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:37:27.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:37:27.294 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:37:27.294 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:37:27.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:37:27.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:37:27.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:37:27.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:37:27.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:37:27.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:37:27.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:37:27.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:37:27.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:37:27.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:37:27.756 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:27.756 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:37:27.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:37:27.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:27.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:37:27.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-02-05T05:37:27.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-02-05T05:37:27.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010' 2024-02-05T05:37:27.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:27.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:37:27.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089349 2024-02-05T05:37:27.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089349 2024-02-05T05:37:27.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349' 2024-02-05T05:37:27.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:27.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:37:28.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383689 2024-02-05T05:37:28.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383689 2024-02-05T05:37:28.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349 2-343597383689' 2024-02-05T05:37:28.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:28.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795010 2024-02-05T05:37:28.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:28.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:37:28.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795010 2024-02-05T05:37:28.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:28.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-02-05T05:37:28.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-02-05T05:37:28.070 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 420906795010 2024-02-05T05:37:28.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:28.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795010 2024-02-05T05:37:28.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:37:29.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:37:29.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:29.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-02-05T05:37:29.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:29.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089349 2024-02-05T05:37:29.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:29.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:37:29.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089349 2024-02-05T05:37:29.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:29.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089349 2024-02-05T05:37:29.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089349' 2024-02-05T05:37:29.606 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 382252089349 2024-02-05T05:37:29.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:37:29.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089349 -lt 382252089349 2024-02-05T05:37:29.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:29.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383689 2024-02-05T05:37:29.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:29.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:37:29.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383689 2024-02-05T05:37:29.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:29.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383689 2024-02-05T05:37:29.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383689' 2024-02-05T05:37:29.879 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 343597383689 2024-02-05T05:37:29.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:37:30.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383689 -lt 343597383689 2024-02-05T05:37:30.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:37:30.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:30.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:30.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:37:30.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:37:30.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:37:30.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:37:30.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:37:30.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:37:30.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:37:30.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:37:30.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:37:30.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:37:30.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:30.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:37:31.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:37:31.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:37:31.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-02-05T05:37:33.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:37:33.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:37:33.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:37:33.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:37:33.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:37:33.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:37:33.414 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:37:33.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:37:33.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:37:33.416 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:37:33.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:37:33.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:37:33.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:37:33.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:37:33.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:37:33.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:37:33.443 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:33.442+0000 7fbd3cc204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:33.445 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:33.444+0000 7fbd3cc204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:33.446 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:33.446+0000 7fbd3cc204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:33.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:33.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:33.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:33.968 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:33.968+0000 7fbd3cc204c0 -1 Falling back to public interface 2024-02-05T05:37:34.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:34.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:34.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:34.960 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:34.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:34.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:35.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:36.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:36.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:37:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:36.310 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:36.309+0000 7fbd3cc204c0 -1 osd.1 100 log_to_monitors true 2024-02-05T05:37:36.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:37.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:37.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:37.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:37:37.519 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:37:37.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:37.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:37:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 105 up_thru 105 down_at 101 last_clean_interval [89,100) [v2:127.0.0.1:6810/2776835563,v1:127.0.0.1:6811/2776835563] [v2:127.0.0.1:6812/2776835563,v1:127.0.0.1:6813/2776835563] exists,up 92637a52-3bf7-4a26-9dfd-8decec4c2bed 2024-02-05T05:37:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:37:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:37:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:37:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:37:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:37:37.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:37:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:37:37.824 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:37:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:37:37.824 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:37:37.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:37:37.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:37:37.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:37:37.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:37:37.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:37:37.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:37:37.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:37:37.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:37:37.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:37:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:37:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:37:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:37:38.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:38.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:37:38.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795013 2024-02-05T05:37:38.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795013 2024-02-05T05:37:38.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013' 2024-02-05T05:37:38.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:38.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:37:38.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-02-05T05:37:38.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-02-05T05:37:38.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-450971566082' 2024-02-05T05:37:38.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:38.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:37:38.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383692 2024-02-05T05:37:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383692 2024-02-05T05:37:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-450971566082 2-343597383692' 2024-02-05T05:37:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:38.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795013 2024-02-05T05:37:38.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:38.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:37:38.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795013 2024-02-05T05:37:38.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:38.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795013 2024-02-05T05:37:38.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795013' 2024-02-05T05:37:38.513 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 420906795013 2024-02-05T05:37:38.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:38.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795013 2024-02-05T05:37:38.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:37:39.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:37:39.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:40.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795013 -lt 420906795013 2024-02-05T05:37:40.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:40.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-450971566082 2024-02-05T05:37:40.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:40.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:37:40.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-450971566082 2024-02-05T05:37:40.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:40.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-02-05T05:37:40.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 450971566082' 2024-02-05T05:37:40.090 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 450971566082 2024-02-05T05:37:40.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:37:40.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-02-05T05:37:40.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:40.357 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383692 2024-02-05T05:37:40.357 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:40.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:37:40.358 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383692 2024-02-05T05:37:40.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:40.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383692 2024-02-05T05:37:40.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383692' 2024-02-05T05:37:40.360 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 343597383692 2024-02-05T05:37:40.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:37:40.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383692 -lt 343597383692 2024-02-05T05:37:40.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:37:40.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:40.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:37:40.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:37:41.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:37:41.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:37:41.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:41.225 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:37:41.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:37:41.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:37:41.918 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:37:41.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:37:42.429 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:37:42.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:37:42.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:37:42.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:37:42.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-02-05T05:37:42.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-02-05T05:37:42.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-02-05T05:37:42.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T05:37:42.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-02-05T05:37:42.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-02-05T05:37:42.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:37:42.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:37:42.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:37:42.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:37:42.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:37:43.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:37:43.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-02-05T05:37:43.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:37:43.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:37:43.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:37:43.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:37:43.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:37:43.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-02-05T05:37:45.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:37:45.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:37:45.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:37:45.080 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:37:45.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:37:45.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:37:45.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:37:45.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:37:45.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:37:45.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:37:45.107 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:45.106+0000 7f00cb68c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:45.109 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:45.108+0000 7f00cb68c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:45.110 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:45.110+0000 7f00cb68c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:45.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:45.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:45.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:45.630 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:45.629+0000 7f00cb68c4c0 -1 Falling back to public interface 2024-02-05T05:37:46.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:46.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:46.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:46.625 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:46.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:46.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:46.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:47.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:47.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:47.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:47.901 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:37:47.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:47.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:47.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:47.962+0000 7f00cb68c4c0 -1 osd.0 109 log_to_monitors true 2024-02-05T05:37:48.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:49.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:49.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:49.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:37:49.174 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:37:49.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:49.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:49.534 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 114 up_thru 114 down_at 110 last_clean_interval [98,109) [v2:127.0.0.1:6802/3633508628,v1:127.0.0.1:6803/3633508628] [v2:127.0.0.1:6804/3633508628,v1:127.0.0.1:6805/3633508628] exists,up 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:37:49.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:37:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:37:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:37:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:37:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:37:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:37:49.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:37:49.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:37:49.901 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:49.901 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:37:49.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:37:49.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:49.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:37:50.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-02-05T05:37:50.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-02-05T05:37:50.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-489626271746' 2024-02-05T05:37:50.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:50.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:37:50.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-02-05T05:37:50.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-02-05T05:37:50.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-489626271746 1-450971566085' 2024-02-05T05:37:50.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:37:50.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:37:50.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383695 2024-02-05T05:37:50.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383695 2024-02-05T05:37:50.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-489626271746 1-450971566085 2-343597383695' 2024-02-05T05:37:50.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:50.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-489626271746 2024-02-05T05:37:50.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:50.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:37:50.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-489626271746 2024-02-05T05:37:50.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:50.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-02-05T05:37:50.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 489626271746' 2024-02-05T05:37:50.311 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 489626271746 2024-02-05T05:37:50.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:50.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 489626271746 2024-02-05T05:37:50.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:37:51.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:37:51.579 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:37:51.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271746 -lt 489626271746 2024-02-05T05:37:51.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:51.857 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-450971566085 2024-02-05T05:37:51.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:51.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:37:51.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-450971566085 2024-02-05T05:37:51.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:51.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-02-05T05:37:51.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 450971566085' 2024-02-05T05:37:51.860 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 450971566085 2024-02-05T05:37:51.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:37:52.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-02-05T05:37:52.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:37:52.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383695 2024-02-05T05:37:52.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:37:52.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:37:52.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383695 2024-02-05T05:37:52.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:37:52.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383695 2024-02-05T05:37:52.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383695' 2024-02-05T05:37:52.130 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 343597383695 2024-02-05T05:37:52.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:37:52.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383695 -lt 343597383695 2024-02-05T05:37:52.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:37:52.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:52.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:52.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:37:52.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:37:52.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:37:52.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:37:52.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:37:52.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:37:52.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:37:52.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:37:52.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:37:52.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:37:52.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:37:52.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:37:53.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:37:53.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:37:53.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:37:53.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:37:53.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:37:53.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:37:53.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:37:53.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:37:53.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:37:53.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-02-05T05:37:56.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:37:56.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:37:56.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:37:56.020 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:37:56.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:37:56.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:37:56.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:37:56.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:37:56.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:37:56.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:37:56.047 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:56.046+0000 7f42174054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:56.049 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:56.048+0000 7f42174054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:56.050 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:56.049+0000 7f42174054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:37:56.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:56.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:56.574 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:56.574+0000 7f42174054c0 -1 Falling back to public interface 2024-02-05T05:37:57.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:57.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:57.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:37:57.565 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:37:57.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:57.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:57.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:37:58.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:37:58.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:37:58.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:37:58.834 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:37:58.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:37:58.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:37:59.002 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:37:59.001+0000 7f42174054c0 -1 osd.0 117 log_to_monitors true 2024-02-05T05:37:59.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:00.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:00.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:00.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:38:00.111 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:38:00.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:00.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:00.563 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 122 up_thru 122 down_at 118 last_clean_interval [114,117) [v2:127.0.0.1:6802/2993158929,v1:127.0.0.1:6803/2993158929] [v2:127.0.0.1:6804/2993158929,v1:127.0.0.1:6805/2993158929] exists,up 8dcd13bb-17e1-4778-a824-1a5028a5e477 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:38:00.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:38:00.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:38:00.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:38:00.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:38:00.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:38:00.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:38:00.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:38:01.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:38:01.032 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:38:01.032 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:38:01.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:38:01.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:38:01.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:38:01.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=523986010114 2024-02-05T05:38:01.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 523986010114 2024-02-05T05:38:01.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114' 2024-02-05T05:38:01.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:38:01.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:38:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-02-05T05:38:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-02-05T05:38:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-450971566088' 2024-02-05T05:38:01.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:38:01.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:38:01.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383698 2024-02-05T05:38:01.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383698 2024-02-05T05:38:01.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-523986010114 1-450971566088 2-343597383698' 2024-02-05T05:38:01.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:38:01.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-523986010114 2024-02-05T05:38:01.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:38:01.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:38:01.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-523986010114 2024-02-05T05:38:01.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:38:01.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=523986010114 2024-02-05T05:38:01.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 523986010114' 2024-02-05T05:38:01.362 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 523986010114 2024-02-05T05:38:01.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:38:01.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 523986010114 2024-02-05T05:38:01.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:38:02.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:38:02.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:38:02.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 523986010114 -lt 523986010114 2024-02-05T05:38:02.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:38:02.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-450971566088 2024-02-05T05:38:02.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:38:02.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:38:02.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-450971566088 2024-02-05T05:38:02.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:38:02.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-02-05T05:38:02.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 450971566088' 2024-02-05T05:38:02.931 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 450971566088 2024-02-05T05:38:02.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:38:03.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566089 -lt 450971566088 2024-02-05T05:38:03.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:38:03.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383698 2024-02-05T05:38:03.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:38:03.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:38:03.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383698 2024-02-05T05:38:03.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:38:03.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383698 2024-02-05T05:38:03.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383698' 2024-02-05T05:38:03.210 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.2 seq 343597383698 2024-02-05T05:38:03.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:38:03.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383699 -lt 343597383698 2024-02-05T05:38:03.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:38:03.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:38:03.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:38:03.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:38:03.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:38:03.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:38:03.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:38:03.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:38:03.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:38:03.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:38:03.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:38:04.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:38:04.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:38:04.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:38:04.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:38:04.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:38:04.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:38:04.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:38:04.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-02-05T05:38:04.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-02-05T05:38:04.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:38:04.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-02-05T05:38:04.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-02-05T05:38:04.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:38:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-02-05T05:38:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-02-05T05:38:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T05:38:04.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T05:38:04.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:04.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:38:04.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:04.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:38:04.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:04.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T05:38:05.306 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-02-05T05:38:05.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:05.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:05.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:38:05.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:35:48.177138+0000 '>' 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:05.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:38:06.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:38:06.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:38:06.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:38:06.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:06.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:38:06.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:06.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:38:06.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:38:06.306150+0000 '>' 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:06.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:38:06.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T05:38:06.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T05:38:06.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-02-05T05:38:06.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T05:38:06.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T05:38:06.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T05:38:06.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T05:38:06.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=122 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:38:06.944 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:38:06.945 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:38:06.945 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:38:06.945 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:38:06.945 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:38:06.945 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:38:06.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:38:06.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:38:06.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:38:06.967 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:38:06.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:38:06.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:38:06.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:38:07.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:07.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T05:38:07.524 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-02-05T05:38:07.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T05:35:48.177138+0000 last_deep_scrub_stamp 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:07.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:38:07.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:35:48.177138+0000 '>' 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:07.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:38:08.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:38:08.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:38:08.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:38:08.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:08.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:38:08.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:08.803 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:38:09.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:35:48.177138+0000 '>' 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:09.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:38:10.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:38:10.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:38:10.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:38:10.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:10.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:38:10.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:10.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:38:10.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:35:48.177138+0000 '>' 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:10.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:38:11.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:38:11.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:38:11.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:38:11.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:11.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:38:11.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:11.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:38:11.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:35:48.177138+0000 '>' 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:11.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:38:12.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:38:12.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:38:12.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:38:12.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:38:12.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:38:12.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:38:12.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:38:12.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:38:08.428328+0000 '>' 2024-02-05T05:35:48.177138+0000 2024-02-05T05:38:12.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:38:12.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-02-05T05:38:12.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-02-05T05:38:12.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-02-05T05:38:12.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T05:38:12.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-02-05T05:38:12.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-02-05T05:38:13.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-02-05T05:38:13.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=122 2024-02-05T05:38:13.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-02-05T05:38:13.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:38:13.044 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:38:13.045 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:38:13.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:38:13.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:38:13.067 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:38:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T05:38:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-02-05T05:38:13.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:38:13.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:38:13.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:38:13.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-02-05T05:38:13.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-02-05T05:38:13.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-02-05T05:38:13.458 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' removed 2024-02-05T05:38:13.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:38:13.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:38:13.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:38:13.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:38:13.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:38:13.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:38:13.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:38:13.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:38:13.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:38:13.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:38:13.579 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:38:13.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:38:13.580 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:38:13.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:38:13.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:38:13.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:38:13.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:38:13.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:38:13.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:38:13.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:38:13.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:38:13.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:38:13.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:38:13.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:38:13.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:13.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:13.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:38:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:38:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:38:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:38:13.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:38:13.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:38:13.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:38:13.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:38:13.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:38:13.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:38:13.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:38:13.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:38:13.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:38:13.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:38:13.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:38:13.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:38:13.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:38:13.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:38:13.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:38:13.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:38:13.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:13.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:13.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:38:13.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:38:13.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:38:13.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:38:13.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:38:13.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:13.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:13.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:38:13.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:38:13.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:38:13.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:38:13.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:38:13.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T05:38:13.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:38:13.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:38:13.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:38:13.992 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:38:13.992 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:13.992 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:13.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:38:13.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:38:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:38:14.162 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:14.162 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:14.162 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:38:14.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:38:14.162 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:38:14.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:38:14.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:38:14.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:38:14.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-02-05T05:38:14.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:38:14.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:38:14.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:38:14.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:38:14.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:38:14.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:38:15.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:38:15.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:38:15.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-02-05T05:38:15.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:38:15.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:38:15.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:38:15.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:38:15.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:38:15.176 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:38:15.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:38:15.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:38:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:38:15.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e7efffd6-654c-4e99-a6be-2f48696bf81a' 2024-02-05T05:38:15.178 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:38:15.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:38:15.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHdMBlMEKACxAAw0b3RQ7ID2TH4C1Wz5MvlA== 2024-02-05T05:38:15.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHdMBlMEKACxAAw0b3RQ7ID2TH4C1Wz5MvlA=="}' 2024-02-05T05:38:15.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7efffd6-654c-4e99-a6be-2f48696bf81a -i td/osd-scrub-repair/0/new.json 2024-02-05T05:38:15.417 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:38:15.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:38:15.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHdMBlMEKACxAAw0b3RQ7ID2TH4C1Wz5MvlA== --osd-uuid e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:38:15.456 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:15.455+0000 7f01f88144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:15.468 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:15.467+0000 7f01f88144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:15.469 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:15.468+0000 7f01f88144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:15.469 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:15.468+0000 7f01f88144c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:38:15.470 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:15.468+0000 7f01f88144c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:38:15.470 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:15.468+0000 7f01f88144c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:38:15.560 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:15.559+0000 7f01f88144c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:38:20.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:38:20.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:38:20.994 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:38:20.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:38:20.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:38:21.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:38:21.339 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:38:21.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:38:21.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:38:21.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:38:21.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:38:21.365 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:21.364+0000 7f9e594184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:21.366 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:21.366+0000 7f9e594184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:21.367 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:21.367+0000 7f9e594184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:21.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:38:21.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:38:21.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:38:21.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:38:21.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:38:21.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:21.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:38:21.609 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:38:21.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:21.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:21.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:21.897 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:21.896+0000 7f9e594184c0 -1 Falling back to public interface 2024-02-05T05:38:22.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:22.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:22.876 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:38:22.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:38:22.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:22.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:23.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:24.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:24.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:24.156 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:38:24.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:38:24.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:24.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:24.243 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:24.241+0000 7f9e594184c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:38:24.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:25.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:25.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:25.456 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:38:25.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:38:25.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:25.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:25.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:26.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:26.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:26.767 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:38:26.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:38:26.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:26.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:27.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:28.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:28.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:28.190 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:38:28.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:38:28.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:28.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:28.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:29.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:29.469 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:38:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:38:29.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:29.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:29.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:30.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:30.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:30.746 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:38:30.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:38:30.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:30.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:31.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:32.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:32.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:32.146 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:38:32.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:38:32.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:32.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:32.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:33.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:33.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:33.424 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:38:33.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:38:33.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:33.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:33.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:34.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:34.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:34.705 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:38:34.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:38:34.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:34.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:35.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:36.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:36.012 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:38:36.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:38:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:36.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:36.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:37.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:37.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:37.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:38:37.559 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:38:37.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:37.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:38.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:39.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:39.129 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:38:39.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:39.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:38:39.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:39.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:39.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:40.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:40.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:40.882 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:38:40.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:38:40.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:40.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:41.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:42.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:42.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:42.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:38:42.316 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:38:42.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:42.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:43.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:44.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:44.070 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:38:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:38:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:44.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:44.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:45.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:45.857 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:38:45.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:45.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:38:45.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:45.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:46.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:47.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:47.268 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:38:47.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:47.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:38:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:47.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:38:47.537 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1870745154,v1:127.0.0.1:6803/1870745154] [v2:127.0.0.1:6804/1870745154,v1:127.0.0.1:6805/1870745154] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:38:47.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:38:47.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:38:47.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:38:47.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:38:47.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:38:47.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:38:47.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 338533ae-6171-4c8a-951a-98411cc8b12b' 2024-02-05T05:38:47.545 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:38:47.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:38:47.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBndMBlgD85IRAAQ/jmKBw1HWX8pV9nON1tBA== 2024-02-05T05:38:47.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBndMBlgD85IRAAQ/jmKBw1HWX8pV9nON1tBA=="}' 2024-02-05T05:38:47.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 338533ae-6171-4c8a-951a-98411cc8b12b -i td/osd-scrub-repair/1/new.json 2024-02-05T05:38:47.877 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:38:47.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:38:47.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBndMBlgD85IRAAQ/jmKBw1HWX8pV9nON1tBA== --osd-uuid 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:38:47.917 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:47.916+0000 7f0d00cc94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:47.962 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:47.961+0000 7f0d00cc94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:47.962 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:47.962+0000 7f0d00cc94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:47.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:47.962+0000 7f0d00cc94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:38:47.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:47.962+0000 7f0d00cc94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:38:47.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:47.962+0000 7f0d00cc94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:38:48.037 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:48.036+0000 7f0d00cc94c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:38:53.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:38:53.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:38:53.755 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:38:53.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:38:53.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:38:54.104 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:38:54.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:38:54.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:38:54.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:38:54.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:38:54.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:38:54.130 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:54.129+0000 7fb27a3fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:54.132 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:54.131+0000 7fb27a3fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:54.133 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:54.132+0000 7fb27a3fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:38:54.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:38:54.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:38:54.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:38:54.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:38:54.410 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:38:54.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:38:54.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:54.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:38:54.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:54.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:38:54.656 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:54.655+0000 7fb27a3fe4c0 -1 Falling back to public interface 2024-02-05T05:38:54.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:55.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:55.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:55.677 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:38:55.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:38:55.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:55.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:38:55.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:56.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:56.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:56.954 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:38:56.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:38:56.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:56.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:38:56.978 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:38:56.977+0000 7fb27a3fe4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:38:57.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:58.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:58.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:58.255 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:38:58.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:38:58.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:58.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:38:58.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:38:59.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:38:59.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:38:59.537 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:38:59.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:38:59.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:38:59.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:38:59.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:00.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:00.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:00.888 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:39:00.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:39:00.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:00.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:01.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:02.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:02.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:02.165 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:39:02.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:39:02.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:02.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:02.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:03.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:03.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:03.443 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:39:03.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:39:03.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:03.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:03.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:04.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:04.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:04.720 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:39:04.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:39:04.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:04.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:05.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:06.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:06.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:06.056 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:39:06.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:39:06.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:06.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:06.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:07.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:07.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:07.335 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:39:07.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:39:07.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:07.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:07.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:08.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:08.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:08.727 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:39:08.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:39:08.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:08.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:09.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:10.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:10.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:10.254 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:39:10.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:39:10.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:10.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:10.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:11.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:11.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:11.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:39:11.842 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:39:11.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:11.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:12.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:13.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:13.563 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:39:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:39:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:13.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:14.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:15.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:15.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:15.204 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:39:15.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:39:15.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:15.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:16.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:17.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:17.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:17.156 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:39:17.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:39:17.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:17.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:18.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:19.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:19.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:19.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:39:19.045 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:39:19.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:19.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:19.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:39:20.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:39:20.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:39:20.386 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:39:20.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:39:20.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:39:20.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:39:20.662 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/33085717,v1:127.0.0.1:6811/33085717] [v2:127.0.0.1:6812/33085717,v1:127.0.0.1:6813/33085717] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:39:20.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:39:20.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:39:20.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:39:20.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-02-05T05:39:20.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:39:20.960 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:39:20.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:39:20.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:39:21.377 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:39:21.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:39:22.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:39:23.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-02-05T05:39:23.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:39:23.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:39:23.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:39:23.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:39:23.973 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:39:23.973 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:39:23.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:39:23.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:39:23.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:39:24.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:39:24.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:39:24.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:39:24.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:39:24.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:39:24.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:39:24.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:39:24.338 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:39:24.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:39:24.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:39:24.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:39:24.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-02-05T05:39:24.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-02-05T05:39:24.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-02-05T05:39:24.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:39:24.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:39:24.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-02-05T05:39:24.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-02-05T05:39:24.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-02-05T05:39:24.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:39:24.549 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-02-05T05:39:24.549 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:39:24.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:39:24.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-02-05T05:39:24.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:39:24.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-02-05T05:39:24.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-02-05T05:39:24.552 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836493 2024-02-05T05:39:24.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:39:24.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-02-05T05:39:24.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:39:25.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:39:25.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:39:26.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-02-05T05:39:26.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:39:26.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-02-05T05:39:26.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:39:26.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:39:26.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-02-05T05:39:26.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:39:26.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-02-05T05:39:26.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-02-05T05:39:26.109 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672965 2024-02-05T05:39:26.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:39:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-02-05T05:39:26.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:39:26.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:39:26.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:39:26.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:39:26.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:39:26.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:39:26.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:39:26.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:39:26.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:39:26.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:39:26.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:39:26.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:39:26.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:39:26.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:39:26.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:39:27.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:39:27.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:39:27.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:39:27.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-02-05T05:39:27.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T05:39:27.719 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'foo' created 2024-02-05T05:39:27.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:39:28.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-02-05T05:39:28.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-02-05T05:39:29.135 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'csr_pool' created 2024-02-05T05:39:29.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:39:30.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-02-05T05:39:30.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:39:30.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:39:30.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:39:30.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:39:30.149 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:39:30.149 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:39:30.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:39:30.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:39:30.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:39:30.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:39:30.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:39:30.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:39:30.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:39:30.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:39:30.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:39:30.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:39:30.515 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:39:30.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:39:30.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:39:30.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:39:30.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-02-05T05:39:30.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-02-05T05:39:30.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-02-05T05:39:30.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:39:30.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:39:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-02-05T05:39:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-02-05T05:39:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672968' 2024-02-05T05:39:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:39:30.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-02-05T05:39:30.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:39:30.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:39:30.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-02-05T05:39:30.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:39:30.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-02-05T05:39:30.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-02-05T05:39:30.730 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836495 2024-02-05T05:39:30.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:39:31.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-02-05T05:39:31.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:39:32.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:39:32.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:39:32.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-02-05T05:39:32.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:39:32.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-02-05T05:39:32.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:39:32.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:39:32.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-02-05T05:39:32.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:39:32.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-02-05T05:39:32.320 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672968 2024-02-05T05:39:32.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-02-05T05:39:32.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:39:32.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-02-05T05:39:32.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:39:32.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:39:32.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:39:32.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:39:32.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:39:32.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:39:32.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:39:32.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:39:32.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:39:32.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:39:32.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:39:33.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:39:33.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:39:33.223 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:39:33.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:39:33.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:39:33.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:39:33.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:39:33.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-02-05T05:39:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-02-05T05:39:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-02-05T05:39:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:33.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-02-05T05:39:33.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:33.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:33.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:33.948 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:33.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:34.350 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:34.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:34.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:34.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:34.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-02-05T05:39:34.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-02-05T05:39:34.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:34.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-02-05T05:39:34.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-02-05T05:39:34.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:34.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:34.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-02-05T05:39:34.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:34.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:34.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:35.338 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:35.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:35.714 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:35.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:35.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:35.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:35.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-02-05T05:39:36.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:36.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:36.610 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:36.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:37.121 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:37.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:37.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-02-05T05:39:37.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-02-05T05:39:37.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:37.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-02-05T05:39:37.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-02-05T05:39:37.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:37.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:37.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-02-05T05:39:37.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:37.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:37.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:38.049 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:38.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:38.552 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:38.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:38.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:38.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:38.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-02-05T05:39:38.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-02-05T05:39:39.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:39.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-02-05T05:39:39.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-02-05T05:39:39.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:39.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:39.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-02-05T05:39:39.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:39.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:39.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:39.482 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:39.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:39.866 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:39.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:39.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:39.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:40.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-02-05T05:39:40.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-02-05T05:39:40.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:40.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-02-05T05:39:40.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-02-05T05:39:40.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:40.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:40.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-02-05T05:39:40.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:40.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:40.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:40.745 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:40.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:41.164 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:41.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:41.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:41.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:41.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-02-05T05:39:41.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-02-05T05:39:41.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:41.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:42.059 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:42.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:42.553 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:42.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:42.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:42.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:42.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-02-05T05:39:42.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-02-05T05:39:43.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:43.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-02-05T05:39:43.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-02-05T05:39:43.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:43.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:43.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-02-05T05:39:43.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:43.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:43.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:43.509 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:43.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:43.885 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:43.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:43.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:43.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:44.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-02-05T05:39:44.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-02-05T05:39:44.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:44.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-02-05T05:39:44.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-02-05T05:39:44.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:44.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:44.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-02-05T05:39:44.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:44.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:44.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:44.763 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:44.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:45.274 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:45.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:45.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:45.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:45.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-02-05T05:39:45.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:45.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:46.154 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:46.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:46.572 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:46.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:46.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:46.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:46.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-02-05T05:39:46.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-02-05T05:39:47.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:47.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-02-05T05:39:47.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-02-05T05:39:47.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:47.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:47.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-02-05T05:39:47.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:47.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:47.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:47.519 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:47.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:47.937 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:48.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-02-05T05:39:48.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-02-05T05:39:48.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:48.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-02-05T05:39:48.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-02-05T05:39:48.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:48.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:48.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-02-05T05:39:48.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:48.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:48.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:48.800 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:48.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:49.210 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:49.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:49.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:49.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:49.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-02-05T05:39:49.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:50.097 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:50.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:50.458 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:50.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:50.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:50.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:50.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-02-05T05:39:50.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:51.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:51.499 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:51.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:51.907 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:51.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:51.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:51.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:52.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-02-05T05:39:52.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-02-05T05:39:52.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:52.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-02-05T05:39:52.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-02-05T05:39:52.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:52.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:52.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-02-05T05:39:52.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:52.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:52.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:52.793 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:52.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:53.262 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:53.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:53.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:53.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:53.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-02-05T05:39:53.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-02-05T05:39:53.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:53.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-02-05T05:39:53.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-02-05T05:39:53.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:53.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:53.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-02-05T05:39:53.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:53.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:53.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:54.158 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:54.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:54.669 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:54.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:54.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:54.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:54.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-02-05T05:39:55.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:55.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:55.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:55.531 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:55.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:56.042 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:56.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:56.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:56.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:56.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-02-05T05:39:56.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:56.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:56.904 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:56.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:57.365 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:57.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:57.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:57.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:57.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-02-05T05:39:57.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-02-05T05:39:57.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:39:57.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-02-05T05:39:57.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-02-05T05:39:57.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:39:57.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:39:57.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-02-05T05:39:57.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:39:57.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:39:57.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:39:58.261 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:39:58.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:39:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:39:58.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:39:58.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:39:58.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:39:58.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-02-05T05:39:59.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-02-05T05:39:59.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-02-05T05:39:59.154 INFO:tasks.workunit.client.0.smithi188.stderr:1025+0 records in 2024-02-05T05:39:59.154 INFO:tasks.workunit.client.0.smithi188.stderr:1025+0 records out 2024-02-05T05:39:59.154 INFO:tasks.workunit.client.0.smithi188.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00180282 s, 582 MB/s 2024-02-05T05:39:59.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-02-05T05:39:59.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-02-05T05:39:59.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-02-05T05:39:59.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-02-05T05:39:59.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-02-05T05:39:59.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-02-05T05:39:59.338 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:39:59.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-02-05T05:39:59.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-02-05T05:39:59.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-02-05T05:39:59.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-02-05T05:39:59.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-02-05T05:39:59.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:39:59.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-02-05T05:39:59.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69197/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-02-05T05:39:59.953 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:39:59.953 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-02-05T05:39:59.953 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:39:59.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-02-05T05:39:59.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:39:59.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:39:59.961 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:39:59.961 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:39:59.961 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:39:59.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:39:59.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-02-05T05:39:59.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.69197/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-02-05T05:40:00.038 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:40:00.038 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-02-05T05:40:00.038 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:40:00.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-02-05T05:40:00.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T05:40:00.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:40:00.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:40:00.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:40:00.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:40:00.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:40:00.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:00.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T05:40:00.561 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T05:39:29.087729+0000 last_deep_scrub_stamp 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:40:00.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:40:00.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:40:00.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:40:00.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:40:00.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:39:29.087729+0000 '>' 2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:00.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:40:01.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:40:01.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:40:01.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:40:01.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:40:01.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:40:01.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:40:01.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:40:02.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:39:29.087729+0000 '>' 2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:02.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:40:03.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:40:03.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:40:03.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:40:03.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:40:03.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:40:03.104 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:40:03.104 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:40:03.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:39:29.087729+0000 '>' 2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:03.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:40:04.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:40:04.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:40:04.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:40:04.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:40:04.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:40:04.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:40:04.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:40:04.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:39:29.087729+0000 '>' 2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:04.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:40:05.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:40:05.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:40:05.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:40:05.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:40:05.649 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:40:05.649 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:40:05.649 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:40:05.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:39:29.087729+0000 '>' 2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:05.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:40:06.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:40:06.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:40:06.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:40:06.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:40:06.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:40:06.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:40:06.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:40:07.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:40:01.806163+0000 '>' 2024-02-05T05:39:29.087729+0000 2024-02-05T05:40:07.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:40:07.195 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-02-05T05:40:07.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:40:07.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-02-05T05:40:07.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-02-05T05:40:07.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:40:07.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:40:07.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:40:07.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:40:07.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:40:07.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:40:07.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:40:07.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:40:07.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:40:07.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:07.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:40:07.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:40:07.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:40:07.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:40:07.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:40:07.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:08.824 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-02-05T05:40:09.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:40:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:40:09.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:40:09.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:40:09.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:40:09.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:40:09.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:40:09.401 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:09.400+0000 7f63a40a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:09.403 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:09.403+0000 7f63a40a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:09.404 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:09.404+0000 7f63a40a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:09.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:09.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:09.927 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:09.927+0000 7f63a40a94c0 -1 Falling back to public interface 2024-02-05T05:40:10.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:10.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:10.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:40:10.917 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:40:10.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:10.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:11.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:12.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:12.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:12.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:40:12.206 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:40:12.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:12.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:12.256 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:12.255+0000 7f63a40a94c0 -1 osd.1 64 log_to_monitors true 2024-02-05T05:40:12.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:13.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:13.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:13.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:40:13.490 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:40:13.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:13.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:13.874 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [10,64) [v2:127.0.0.1:6810/258160396,v1:127.0.0.1:6811/258160396] [v2:127.0.0.1:6812/258160396,v1:127.0.0.1:6813/258160396] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:40:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:40:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:40:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:40:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:40:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:40:13.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:40:13.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:40:13.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:40:13.876 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:40:13.876 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:40:13.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:40:13.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:40:13.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:40:13.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:40:13.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:40:13.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:40:13.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:40:13.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:40:13.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:40:14.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:40:14.230 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:40:14.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:40:14.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:14.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:40:14.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-05T05:40:14.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-05T05:40:14.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-05T05:40:14.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:14.342 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:40:14.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776130 2024-02-05T05:40:14.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776130 2024-02-05T05:40:14.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-292057776130' 2024-02-05T05:40:14.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:14.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-05T05:40:14.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:14.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:40:14.447 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-05T05:40:14.447 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:14.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-05T05:40:14.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-05T05:40:14.448 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836505 2024-02-05T05:40:14.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:14.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-02-05T05:40:14.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:40:15.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:40:15.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:15.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-02-05T05:40:15.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:15.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776130 2024-02-05T05:40:15.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:15.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:40:15.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776130 2024-02-05T05:40:15.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:15.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776130 2024-02-05T05:40:15.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2024-02-05T05:40:15.992 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 292057776130 2024-02-05T05:40:15.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:40:16.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776130 -lt 292057776130 2024-02-05T05:40:16.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:40:16.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:16.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:40:16.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:40:16.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:40:16.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:40:16.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:40:16.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:40:16.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:40:16.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:40:16.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:40:16.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:40:16.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:16.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:17.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:40:17.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:40:17.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:40:17.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:40:17.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-02-05T05:40:17.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:40:17.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:40:17.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:40:17.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:40:17.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:17.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:40:17.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:40:17.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:40:17.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:40:17.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:40:17.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:19.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:40:19.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:40:19.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:40:19.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:40:19.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:40:19.283 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:40:19.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:40:19.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:40:19.286 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:40:19.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:40:19.286 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:40:19.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:40:19.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:40:19.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:40:19.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:40:19.313 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:19.312+0000 7fabfb4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:19.328 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:19.327+0000 7fabfb4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:19.329 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:19.328+0000 7fabfb4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:19.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:19.850 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:19.849+0000 7fabfb4144c0 -1 Falling back to public interface 2024-02-05T05:40:19.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:20.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:20.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:20.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:40:20.903 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:40:20.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:20.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:21.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:22.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:22.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:22.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:40:22.186 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:40:22.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:22.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:22.203 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:22.202+0000 7fabfb4144c0 -1 osd.0 70 log_to_monitors true 2024-02-05T05:40:22.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:23.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:23.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:23.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:40:23.565 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:40:23.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:23.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 74 up_thru 74 down_at 71 last_clean_interval [5,70) [v2:127.0.0.1:6802/709868043,v1:127.0.0.1:6803/709868043] [v2:127.0.0.1:6804/709868043,v1:127.0.0.1:6805/709868043] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:40:23.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:40:23.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:40:23.845 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:40:23.845 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:40:23.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:40:23.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:40:23.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:40:23.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:40:23.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:40:23.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:40:23.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:40:23.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:40:23.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:40:24.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:40:24.199 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:40:24.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:40:24.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:24.199 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:40:24.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-02-05T05:40:24.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-02-05T05:40:24.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579906' 2024-02-05T05:40:24.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:24.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:40:24.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776133 2024-02-05T05:40:24.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776133 2024-02-05T05:40:24.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579906 1-292057776133' 2024-02-05T05:40:24.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:24.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-317827579906 2024-02-05T05:40:24.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:24.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:40:24.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-317827579906 2024-02-05T05:40:24.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:24.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-02-05T05:40:24.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 317827579906' 2024-02-05T05:40:24.411 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 317827579906 2024-02-05T05:40:24.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:24.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 317827579906 2024-02-05T05:40:24.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:40:25.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:40:25.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:25.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-02-05T05:40:25.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:25.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-292057776133 2024-02-05T05:40:25.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:25.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:40:25.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-292057776133 2024-02-05T05:40:25.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:25.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776133 2024-02-05T05:40:25.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 292057776133' 2024-02-05T05:40:25.956 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 292057776133 2024-02-05T05:40:25.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:40:26.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776133 -lt 292057776133 2024-02-05T05:40:26.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:40:26.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:26.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:26.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:40:26.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:40:26.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:40:26.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:40:26.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:40:26.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:40:26.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:40:26.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:40:26.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:40:26.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:40:26.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:26.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:27.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:40:27.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:40:27.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:40:27.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:40:27.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-02-05T05:40:27.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-02-05T05:40:27.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:40:27.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:40:27.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:40:27.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:40:27.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:40:27.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:40:27.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:40:27.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-02-05T05:40:27.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:40:27.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:40:27.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:40:27.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:40:27.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:40:27.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-02-05T05:40:28.640 INFO:tasks.workunit.client.0.smithi188.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-02-05T05:40:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:40:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:40:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:40:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:40:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:40:29.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:40:29.163 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:40:29.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:40:29.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:40:29.165 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:40:29.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:40:29.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:40:29.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:40:29.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:40:29.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:40:29.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:40:29.191 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:29.190+0000 7fc3b62104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:29.193 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:29.193+0000 7fc3b62104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:29.195 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:29.194+0000 7fc3b62104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:29.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:40:29.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:40:29.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:40:29.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:40:29.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:40:29.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:29.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:40:29.441 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:40:29.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:29.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:29.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:29.716 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:29.716+0000 7fc3b62104c0 -1 Falling back to public interface 2024-02-05T05:40:30.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:30.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:30.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:40:30.711 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:40:30.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:30.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:30.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:31.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:31.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:31.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:40:31.991 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:40:31.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:31.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:32.072 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:32.071+0000 7fc3b62104c0 -1 osd.1 75 log_to_monitors true 2024-02-05T05:40:32.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:33.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:33.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:33.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:40:33.268 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:40:33.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:33.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [68,75) [v2:127.0.0.1:6810/4054599911,v1:127.0.0.1:6811/4054599911] [v2:127.0.0.1:6812/4054599911,v1:127.0.0.1:6813/4054599911] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:40:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:40:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:40:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:40:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:40:33.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:40:33.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:40:33.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:40:33.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:40:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:40:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:40:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:40:33.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:40:33.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:40:33.933 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:40:33.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:40:33.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:33.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:40:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579909 2024-02-05T05:40:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579909 2024-02-05T05:40:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579909' 2024-02-05T05:40:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:34.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:40:34.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416386 2024-02-05T05:40:34.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416386 2024-02-05T05:40:34.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-317827579909 1-339302416386' 2024-02-05T05:40:34.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:34.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-317827579909 2024-02-05T05:40:34.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:34.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:40:34.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-317827579909 2024-02-05T05:40:34.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:34.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579909 2024-02-05T05:40:34.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 317827579909' 2024-02-05T05:40:34.143 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 317827579909 2024-02-05T05:40:34.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:34.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579908 -lt 317827579909 2024-02-05T05:40:34.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:40:35.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:40:35.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:35.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579909 -lt 317827579909 2024-02-05T05:40:35.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:35.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416386 2024-02-05T05:40:35.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:35.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:40:35.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416386 2024-02-05T05:40:35.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:35.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416386 2024-02-05T05:40:35.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2024-02-05T05:40:35.684 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 339302416386 2024-02-05T05:40:35.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:40:35.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416386 -lt 339302416386 2024-02-05T05:40:35.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:40:35.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:35.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:40:36.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:40:36.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:40:36.578 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:40:36.578 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:36.578 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:36.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:40:36.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:40:36.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:40:36.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:40:36.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-02-05T05:40:36.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-02-05T05:40:36.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:40:36.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:40:36.913 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:40:36.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:40:36.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:40:36.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:40:37.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-02-05T05:40:38.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:40:38.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:40:38.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:40:38.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:40:38.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:40:38.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:40:38.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:40:38.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:40:38.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:40:38.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:40:38.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:40:38.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:40:38.838 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:40:38.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:40:38.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:40:38.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:40:38.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:40:38.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:40:38.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:40:38.863 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:38.862+0000 7f53e1b274c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:38.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:38.864+0000 7f53e1b274c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:38.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:38.866+0000 7f53e1b274c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:39.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:39.381 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:39.380+0000 7f53e1b274c0 -1 Falling back to public interface 2024-02-05T05:40:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:40.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:40.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:40.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:40:40.406 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:40:40.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:40.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:40.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:41.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:41.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:41.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:40:41.683 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:40:41.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:41.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:41.781 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:41.780+0000 7f53e1b274c0 -1 osd.0 80 log_to_monitors true 2024-02-05T05:40:41.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:42.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:42.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:42.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:40:42.961 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:40:42.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:42.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [74,80) [v2:127.0.0.1:6802/1675270541,v1:127.0.0.1:6803/1675270541] [v2:127.0.0.1:6804/1675270541,v1:127.0.0.1:6805/1675270541] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:40:43.390 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:40:43.391 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:40:43.391 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:40:43.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:40:43.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:40:43.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:40:43.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:40:43.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:40:43.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:40:43.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:40:43.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:40:43.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:40:43.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:40:43.743 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:40:43.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:40:43.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:43.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:40:43.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-02-05T05:40:43.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-02-05T05:40:43.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866' 2024-02-05T05:40:43.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:43.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:40:43.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=339302416389 2024-02-05T05:40:43.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 339302416389 2024-02-05T05:40:43.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252866 1-339302416389' 2024-02-05T05:40:43.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:43.950 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252866 2024-02-05T05:40:43.950 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:43.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:40:43.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252866 2024-02-05T05:40:43.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:43.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-02-05T05:40:43.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2024-02-05T05:40:43.953 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 360777252866 2024-02-05T05:40:43.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:44.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 360777252866 2024-02-05T05:40:44.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:40:45.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:40:45.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:45.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-02-05T05:40:45.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:45.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-339302416389 2024-02-05T05:40:45.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:45.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:40:45.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-339302416389 2024-02-05T05:40:45.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:45.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=339302416389 2024-02-05T05:40:45.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 339302416389' 2024-02-05T05:40:45.510 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 339302416389 2024-02-05T05:40:45.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:40:45.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 339302416389 -lt 339302416389 2024-02-05T05:40:45.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:40:45.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:45.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:46.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:40:46.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:40:46.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:40:46.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:40:46.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:40:46.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:40:46.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:40:46.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:40:46.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:40:46.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:40:46.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:46.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:46.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:40:46.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:40:46.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:40:46.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:40:46.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-02-05T05:40:46.740 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:40:46.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:40:46.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:40:46.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:40:46.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:40:46.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:40:46.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:40:47.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:40:47.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T05:40:47.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:40:47.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:40:47.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:40:47.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:40:47.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:40:47.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-02-05T05:40:48.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:40:48.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:40:48.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:40:48.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:40:48.795 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:40:48.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:40:48.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:40:48.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:40:48.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:40:48.797 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:40:48.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:40:48.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:40:48.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:40:48.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:40:48.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:40:48.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:40:48.823 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:48.823+0000 7f0680a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:48.826 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:48.825+0000 7f0680a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:48.827 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:48.826+0000 7f0680a144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:49.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:49.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:49.346 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:49.345+0000 7f0680a144c0 -1 Falling back to public interface 2024-02-05T05:40:49.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:50.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:50.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:50.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:40:50.384 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:40:50.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:50.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:50.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:51.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:51.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:51.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:40:51.660 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:40:51.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:51.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:51.797 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:51.796+0000 7f0680a144c0 -1 osd.1 86 log_to_monitors true 2024-02-05T05:40:51.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:52.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:40:52.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:52.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:40:52.935 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:40:52.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:52.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:40:53.220 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 87 last_clean_interval [79,86) [v2:127.0.0.1:6810/3120502148,v1:127.0.0.1:6811/3120502148] [v2:127.0.0.1:6812/3120502148,v1:127.0.0.1:6813/3120502148] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:40:53.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:40:53.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:40:53.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:40:53.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:40:53.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:40:53.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:40:53.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:40:53.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:40:53.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:40:53.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:40:53.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:40:53.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:40:53.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:40:53.574 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:40:53.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:40:53.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:53.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:40:53.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252869 2024-02-05T05:40:53.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252869 2024-02-05T05:40:53.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869' 2024-02-05T05:40:53.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:40:53.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:40:53.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056642 2024-02-05T05:40:53.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056642 2024-02-05T05:40:53.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-360777252869 1-386547056642' 2024-02-05T05:40:53.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:53.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-360777252869 2024-02-05T05:40:53.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:53.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:40:53.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-360777252869 2024-02-05T05:40:53.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:53.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252869 2024-02-05T05:40:53.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 360777252869' 2024-02-05T05:40:53.784 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 360777252869 2024-02-05T05:40:53.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:54.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252868 -lt 360777252869 2024-02-05T05:40:54.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:40:55.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:40:55.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:40:55.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252869 -lt 360777252869 2024-02-05T05:40:55.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:40:55.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056642 2024-02-05T05:40:55.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:40:55.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:40:55.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056642 2024-02-05T05:40:55.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:40:55.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056642 2024-02-05T05:40:55.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2024-02-05T05:40:55.385 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 386547056642 2024-02-05T05:40:55.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:40:55.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056642 -lt 386547056642 2024-02-05T05:40:55.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:40:55.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:55.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:55.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:40:55.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:40:55.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:40:55.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:40:55.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:40:55.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:40:55.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:40:55.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:40:56.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:40:56.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:40:56.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:40:56.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:40:56.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:40:56.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:40:56.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:40:56.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:40:56.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-02-05T05:40:56.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:40:56.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:40:56.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:40:56.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:40:56.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T05:40:56.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:40:56.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:40:56.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:40:56.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:40:56.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:40:56.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-02-05T05:40:58.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:40:58.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:40:58.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:40:58.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:40:58.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:40:58.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:40:58.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:40:58.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:40:58.643 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:40:58.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:40:58.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:40:58.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:40:58.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:40:58.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:40:58.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:40:58.670 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:58.669+0000 7ff246e984c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:58.676 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:58.672+0000 7ff246e984c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:58.676 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:58.673+0000 7ff246e984c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:40:58.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:40:58.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:40:59.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:40:59.198 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:40:59.197+0000 7ff246e984c0 -1 Falling back to public interface 2024-02-05T05:41:00.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:00.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:00.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:00.191 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:41:00.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:00.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:41:00.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:01.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:01.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:01.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:01.473 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:41:01.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:01.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:41:01.685 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:01.684+0000 7ff246e984c0 -1 osd.0 91 log_to_monitors true 2024-02-05T05:41:01.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:02.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:02.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:02.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:02.747 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:41:02.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:02.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:41:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6802/2740180248,v1:127.0.0.1:6803/2740180248] [v2:127.0.0.1:6804/2740180248,v1:127.0.0.1:6805/2740180248] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:41:03.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:03.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:03.103 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:03.103 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:03.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:03.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:03.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:03.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:03.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:03.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:03.475 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:41:03.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:03.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:03.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:03.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893122 2024-02-05T05:41:03.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893122 2024-02-05T05:41:03.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122' 2024-02-05T05:41:03.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:03.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:03.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=386547056645 2024-02-05T05:41:03.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 386547056645 2024-02-05T05:41:03.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893122 1-386547056645' 2024-02-05T05:41:03.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:03.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893122 2024-02-05T05:41:03.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:03.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:03.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893122 2024-02-05T05:41:03.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:03.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893122 2024-02-05T05:41:03.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2024-02-05T05:41:03.697 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 408021893122 2024-02-05T05:41:03.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:03.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 408021893122 2024-02-05T05:41:03.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:04.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:04.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:05.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893122 -lt 408021893122 2024-02-05T05:41:05.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:05.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-386547056645 2024-02-05T05:41:05.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:05.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:05.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-386547056645 2024-02-05T05:41:05.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:05.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=386547056645 2024-02-05T05:41:05.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 386547056645' 2024-02-05T05:41:05.250 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 386547056645 2024-02-05T05:41:05.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:05.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 386547056645 -lt 386547056645 2024-02-05T05:41:05.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:05.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:05.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:05.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:06.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:41:06.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:06.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:06.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:06.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:41:06.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:06.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:06.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-02-05T05:41:06.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:41:06.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-02-05T05:41:06.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:41:06.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:06.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:41:06.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:06.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:06.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:06.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:06.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:06.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:08.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:41:08.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:08.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:41:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:41:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:41:08.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:08.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:41:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:41:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:08.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:08.409 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:08.408+0000 7fe17cc144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:08.411 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:08.411+0000 7fe17cc144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:08.413 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:08.412+0000 7fe17cc144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:41:08.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:08.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:08.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:08.929 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:08.928+0000 7fe17cc144c0 -1 Falling back to public interface 2024-02-05T05:41:09.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:09.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:09.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:09.925 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:41:09.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:09.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:10.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:11.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:11.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:11.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:11.200 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:41:11.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:11.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:11.321 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:11.320+0000 7fe17cc144c0 -1 osd.1 97 log_to_monitors true 2024-02-05T05:41:11.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:12.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:12.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:12.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:12.535 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:41:12.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:12.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:12.812 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/2982276975,v1:127.0.0.1:6811/2982276975] [v2:127.0.0.1:6812/2982276975,v1:127.0.0.1:6813/2982276975] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:41:12.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:12.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:12.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:12.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:12.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:12.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:12.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:12.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:13.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:13.165 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:41:13.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:13.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:13.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:13.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=408021893125 2024-02-05T05:41:13.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 408021893125 2024-02-05T05:41:13.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893125' 2024-02-05T05:41:13.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:13.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:13.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-02-05T05:41:13.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-02-05T05:41:13.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-408021893125 1-433791696898' 2024-02-05T05:41:13.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:13.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-408021893125 2024-02-05T05:41:13.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:13.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:13.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-408021893125 2024-02-05T05:41:13.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:13.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=408021893125 2024-02-05T05:41:13.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 408021893125' 2024-02-05T05:41:13.374 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 408021893125 2024-02-05T05:41:13.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:13.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893125 2024-02-05T05:41:13.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:14.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:14.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:14.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893124 -lt 408021893125 2024-02-05T05:41:14.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:15.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:41:15.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:16.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 408021893125 -lt 408021893125 2024-02-05T05:41:16.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:16.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696898 2024-02-05T05:41:16.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:16.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:16.195 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696898 2024-02-05T05:41:16.195 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:16.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-02-05T05:41:16.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2024-02-05T05:41:16.196 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 433791696898 2024-02-05T05:41:16.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:16.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-02-05T05:41:16.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:16.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:16.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:16.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:17.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:41:17.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:17.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:17.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:17.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:41:17.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:17.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:17.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-02-05T05:41:17.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:41:17.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-02-05T05:41:17.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-02-05T05:41:17.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:41:17.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:41:17.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-02-05T05:41:17.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:41:17.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:17.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:41:17.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:17.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:17.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:17.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:17.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:17.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T05:41:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:41:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:41:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-02-05T05:41:19.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:41:19.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:19.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:19.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:19.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:41:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:41:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:41:19.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:19.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:41:19.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:41:19.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:19.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:19.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:19.323 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:19.322+0000 7fb951c674c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:19.325 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:19.324+0000 7fb951c674c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:19.326 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:19.326+0000 7fb951c674c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:41:19.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:19.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:41:19.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:19.846 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:19.846+0000 7fb951c674c0 -1 Falling back to public interface 2024-02-05T05:41:20.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:20.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:20.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:20.839 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:41:20.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:20.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:41:21.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:22.108 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:22.107+0000 7fb951c674c0 -1 osd.0 102 log_to_monitors true 2024-02-05T05:41:22.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:22.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:22.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:22.113 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:41:22.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:22.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:41:22.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:41:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:23.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:41:23.739 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6802/3937626950,v1:127.0.0.1:6803/3937626950] [v2:127.0.0.1:6804/3937626950,v1:127.0.0.1:6805/3937626950] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:41:23.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:23.740 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:23.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:23.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:23.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:23.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:23.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:23.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:24.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:24.093 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:41:24.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:24.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:24.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-02-05T05:41:24.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-02-05T05:41:24.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378' 2024-02-05T05:41:24.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:24.195 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:24.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-02-05T05:41:24.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-02-05T05:41:24.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533378 1-433791696901' 2024-02-05T05:41:24.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:24.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533378 2024-02-05T05:41:24.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:24.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:24.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533378 2024-02-05T05:41:24.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:24.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-02-05T05:41:24.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2024-02-05T05:41:24.301 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 455266533378 2024-02-05T05:41:24.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:24.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 455266533378 2024-02-05T05:41:24.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:25.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:25.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:25.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-02-05T05:41:25.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:25.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-433791696901 2024-02-05T05:41:25.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:25.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:25.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-433791696901 2024-02-05T05:41:25.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:25.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-02-05T05:41:25.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 433791696901' 2024-02-05T05:41:25.846 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 433791696901 2024-02-05T05:41:25.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:26.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696901 -lt 433791696901 2024-02-05T05:41:26.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:26.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:26.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:26.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:26.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:41:26.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:26.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:26.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:27.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:41:27.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:27.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:27.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:27.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:27.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:27.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:27.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:27.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T05:41:27.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:27.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:27.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:41:27.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:27.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:27.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-02-05T05:41:28.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:41:28.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:28.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:28.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:41:28.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:28.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:28.913 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:28.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:41:28.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:41:28.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:28.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:28.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:28.940 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:28.939+0000 7fa6f0a164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:28.943 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:28.942+0000 7fa6f0a164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:28.944 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:28.943+0000 7fa6f0a164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:29.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:29.462 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:29.461+0000 7fa6f0a164c0 -1 Falling back to public interface 2024-02-05T05:41:29.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:30.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:30.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:30.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:30.506 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:41:30.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:30.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:31.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:31.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:31.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:31.784 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:41:31.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:31.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:31.923 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:31.922+0000 7fa6f0a164c0 -1 osd.1 108 log_to_monitors true 2024-02-05T05:41:32.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:33.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:33.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:33.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:33.066 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:41:33.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:33.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:33.447 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 112 up_thru 112 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/3584698030,v1:127.0.0.1:6811/3584698030] [v2:127.0.0.1:6812/3584698030,v1:127.0.0.1:6813/3584698030] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:41:33.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:33.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:33.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:33.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:33.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:33.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:33.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:33.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:33.449 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:33.449 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:33.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:33.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:33.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:33.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:33.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:33.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:33.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:33.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:33.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:33.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:33.843 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:41:33.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:33.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:33.843 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:33.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533381 2024-02-05T05:41:33.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533381 2024-02-05T05:41:33.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533381' 2024-02-05T05:41:33.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:33.988 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:34.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=481036337154 2024-02-05T05:41:34.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 481036337154 2024-02-05T05:41:34.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533381 1-481036337154' 2024-02-05T05:41:34.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:34.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533381 2024-02-05T05:41:34.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:34.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:34.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533381 2024-02-05T05:41:34.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:34.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533381 2024-02-05T05:41:34.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533381' 2024-02-05T05:41:34.093 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 455266533381 2024-02-05T05:41:34.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:34.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533380 -lt 455266533381 2024-02-05T05:41:34.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:35.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:35.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:35.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533381 -lt 455266533381 2024-02-05T05:41:35.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:35.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-481036337154 2024-02-05T05:41:35.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:35.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:35.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-481036337154 2024-02-05T05:41:35.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:35.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=481036337154 2024-02-05T05:41:35.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2024-02-05T05:41:35.638 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 481036337154 2024-02-05T05:41:35.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 481036337154 -lt 481036337154 2024-02-05T05:41:35.904 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:35.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:35.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:36.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:41:36.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:36.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:36.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:36.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:36.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:36.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:36.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:36.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:41:36.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:36.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:36.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:36.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:37.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:37.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T05:41:37.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:37.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:37.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:41:37.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:37.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:37.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-02-05T05:41:38.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:41:38.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:38.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:38.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:38.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:38.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:38.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:38.683 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:38.682+0000 7f3f6640c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:38.685 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:38.684+0000 7f3f6640c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:38.686 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:38.686+0000 7f3f6640c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:38.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:41:38.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:38.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:41:38.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:38.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:38.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:38.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:38.931 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:41:38.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:38.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:39.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:39.208 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:39.208+0000 7f3f6640c4c0 -1 Falling back to public interface 2024-02-05T05:41:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:41:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:40.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:41.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:41.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:41.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:41.477 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:41:41.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:41.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:41.594 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:41.592+0000 7f3f6640c4c0 -1 osd.1 113 log_to_monitors true 2024-02-05T05:41:41.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:42.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:42.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:42.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:42.764 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:41:42.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:42.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:43.048 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 117 up_thru 117 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/4209450059,v1:127.0.0.1:6811/4209450059] [v2:127.0.0.1:6812/4209450059,v1:127.0.0.1:6813/4209450059] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:41:43.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:43.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:43.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:43.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:43.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:43.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:43.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:43.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:43.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:43.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:41:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:43.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:43.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533384 2024-02-05T05:41:43.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533384 2024-02-05T05:41:43.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533384' 2024-02-05T05:41:43.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:43.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:43.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-02-05T05:41:43.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-02-05T05:41:43.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533384 1-502511173634' 2024-02-05T05:41:43.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:43.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533384 2024-02-05T05:41:43.616 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:43.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:43.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533384 2024-02-05T05:41:43.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:43.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533384 2024-02-05T05:41:43.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533384' 2024-02-05T05:41:43.619 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 455266533384 2024-02-05T05:41:43.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:43.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533383 -lt 455266533384 2024-02-05T05:41:43.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:44.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:44.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:45.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533384 -lt 455266533384 2024-02-05T05:41:45.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:45.170 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-502511173634 2024-02-05T05:41:45.170 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:45.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:45.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-502511173634 2024-02-05T05:41:45.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:45.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-02-05T05:41:45.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-02-05T05:41:45.173 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 502511173634 2024-02-05T05:41:45.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:45.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-02-05T05:41:45.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:45.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:45.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:45.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:41:45.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:45.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:45.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:45.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:45.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:45.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:45.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:46.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:41:46.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:46.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:46.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:46.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:41:46.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:46.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:46.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-02-05T05:41:46.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:41:46.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-02-05T05:41:46.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-02-05T05:41:46.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:46.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:46.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:46.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:46.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:46.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:46.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-02-05T05:41:46.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:46.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:46.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:41:46.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:46.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:46.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-02-05T05:41:48.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:41:48.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:48.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:41:48.412 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:41:48.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:48.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:41:48.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:41:48.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:48.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:48.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:48.439 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:48.438+0000 7f2434d1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:48.441 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:48.440+0000 7f2434d1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:48.442 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:48.442+0000 7f2434d1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:48.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:48.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:48.959 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:48.958+0000 7f2434d1a4c0 -1 Falling back to public interface 2024-02-05T05:41:49.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:49.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:49.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:41:49.958 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:41:49.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:49.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:50.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:51.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:51.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:51.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:41:51.234 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:41:51.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:51.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:51.373 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:51.372+0000 7f2434d1a4c0 -1 osd.1 119 log_to_monitors true 2024-02-05T05:41:51.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:41:52.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:41:52.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:52.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:41:52.506 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:41:52.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:52.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 123 up_thru 123 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/3606266481,v1:127.0.0.1:6811/3606266481] [v2:127.0.0.1:6812/3606266481,v1:127.0.0.1:6813/3606266481] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:41:52.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:41:52.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:41:52.808 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:41:52.808 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:41:52.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:41:52.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:41:52.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:41:52.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:41:52.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:41:52.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:41:52.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:41:52.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:41:52.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:41:53.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:41:53.159 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:41:53.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:41:53.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:53.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:41:53.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533387 2024-02-05T05:41:53.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533387 2024-02-05T05:41:53.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533387' 2024-02-05T05:41:53.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:41:53.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:41:53.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-02-05T05:41:53.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-02-05T05:41:53.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533387 1-528280977410' 2024-02-05T05:41:53.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:53.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533387 2024-02-05T05:41:53.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:53.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:41:53.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533387 2024-02-05T05:41:53.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:53.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533387 2024-02-05T05:41:53.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533387' 2024-02-05T05:41:53.377 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 455266533387 2024-02-05T05:41:53.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:53.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533386 -lt 455266533387 2024-02-05T05:41:53.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:54.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:41:54.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:54.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533386 -lt 455266533387 2024-02-05T05:41:54.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:41:55.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:41:55.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:41:56.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533387 -lt 455266533387 2024-02-05T05:41:56.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:41:56.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-528280977410 2024-02-05T05:41:56.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:41:56.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:41:56.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-528280977410 2024-02-05T05:41:56.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:41:56.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-02-05T05:41:56.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-02-05T05:41:56.192 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 528280977410 2024-02-05T05:41:56.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:41:56.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-02-05T05:41:56.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:41:56.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:56.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:41:56.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:41:57.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:41:57.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:41:57.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:41:57.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:41:57.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:41:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:41:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:41:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-02-05T05:41:57.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-02-05T05:41:57.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:41:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:41:57.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:41:57.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:41:57.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:41:57.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:41:57.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:41:57.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:41:57.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:41:57.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:41:57.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:57.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:41:59.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:41:59.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:41:59.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:41:59.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:41:59.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:41:59.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:41:59.620 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:41:59.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:41:59.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:41:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:41:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:41:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:41:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:41:59.647 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:59.646+0000 7fc5680134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:59.649 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:59.648+0000 7fc5680134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:59.650 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:41:59.650+0000 7fc5680134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:41:59.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:41:59.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:41:59.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:41:59.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:41:59.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:41:59.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:41:59.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:41:59.896 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:41:59.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:41:59.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:00.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:00.173 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:00.172+0000 7fc5680134c0 -1 Falling back to public interface 2024-02-05T05:42:01.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:01.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:01.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:42:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:01.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:02.415 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:02.415+0000 7fc5680134c0 -1 osd.1 124 log_to_monitors true 2024-02-05T05:42:02.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:42:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:02.796 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 128 up_thru 123 down_at 125 last_clean_interval [123,124) [v2:127.0.0.1:6810/3937275907,v1:127.0.0.1:6811/3937275907] [v2:127.0.0.1:6812/3937275907,v1:127.0.0.1:6813/3937275907] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:42:02.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:02.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:02.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:02.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:02.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:02.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:02.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:02.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:02.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:02.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:02.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:03.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:03.159 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:42:03.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:03.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:03.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:03.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533390 2024-02-05T05:42:03.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533390 2024-02-05T05:42:03.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533390' 2024-02-05T05:42:03.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:03.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:03.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=549755813890 2024-02-05T05:42:03.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 549755813890 2024-02-05T05:42:03.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-455266533390 1-549755813890' 2024-02-05T05:42:03.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:03.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-455266533390 2024-02-05T05:42:03.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:03.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:03.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-455266533390 2024-02-05T05:42:03.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:03.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533390 2024-02-05T05:42:03.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 455266533390' 2024-02-05T05:42:03.406 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 455266533390 2024-02-05T05:42:03.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:03.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533389 -lt 455266533390 2024-02-05T05:42:03.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:04.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:42:04.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:05.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533389 -lt 455266533390 2024-02-05T05:42:05.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:06.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:42:06.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:06.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533390 -lt 455266533390 2024-02-05T05:42:06.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:06.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-549755813890 2024-02-05T05:42:06.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:06.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:06.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-549755813890 2024-02-05T05:42:06.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:06.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=549755813890 2024-02-05T05:42:06.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 549755813890' 2024-02-05T05:42:06.355 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 549755813890 2024-02-05T05:42:06.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:06.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 549755813890 -lt 549755813890 2024-02-05T05:42:06.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:06.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:06.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:07.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:42:07.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:07.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:07.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:07.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:07.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:07.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:07.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:07.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:42:07.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:07.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:07.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:07.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-02-05T05:42:07.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T05:42:07.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:42:07.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T05:42:07.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:42:07.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:42:07.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:42:07.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:07.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:07.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:07.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:07.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:07.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:07.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:08.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:42:08.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:08.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:08.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 6 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:42:08.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:42:08.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:42:08.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:42:08.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:42:08.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:42:08.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:42:08.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:08.892 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:09.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:42:09.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:09.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:09.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:09.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:42:09.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:09.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:09.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-02-05T05:42:09.489 INFO:tasks.workunit.client.0.smithi188.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-02-05T05:42:09.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:09.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-02-05T05:42:09.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-02-05T05:42:09.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:42:09.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:09.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:09.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-02-05T05:42:09.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-02-05T05:42:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:42:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:09.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-02-05T05:42:09.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-02-05T05:42:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:42:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:09.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-02-05T05:42:09.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-02-05T05:42:09.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:42:09.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:09.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:09.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-02-05T05:42:09.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-02-05T05:42:09.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:42:09.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:09.498 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:09.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:09.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:09.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:09.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:09.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T05:42:09.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:09.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:09.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:42:09.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:09.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:42:09.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-02-05T05:42:11.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:11.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:11.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:42:11.419 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:42:11.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:11.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:42:11.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:42:11.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:11.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:11.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:11.443 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:11.443+0000 7f0d81b724c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:11.448 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:11.445+0000 7f0d81b724c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:11.448 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:11.446+0000 7f0d81b724c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:11.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:11.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:11.965 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:11.965+0000 7f0d81b724c0 -1 Falling back to public interface 2024-02-05T05:42:12.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:12.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:12.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:12.959 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:42:12.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:12.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:13.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:14.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:14.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:14.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:14.229 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:42:14.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:14.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:14.315 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:14.313+0000 7f0d81b724c0 -1 osd.0 130 log_to_monitors true 2024-02-05T05:42:14.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:42:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:15.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 134 up_thru 134 down_at 131 last_clean_interval [106,130) [v2:127.0.0.1:6802/3742508663,v1:127.0.0.1:6803/3742508663] [v2:127.0.0.1:6804/3742508663,v1:127.0.0.1:6805/3742508663] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:15.897 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:15.898 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:15.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:15.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:15.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:15.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:15.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:15.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:15.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:15.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:15.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:16.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:16.253 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:42:16.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:16.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:16.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:16.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=575525617666 2024-02-05T05:42:16.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 575525617666 2024-02-05T05:42:16.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617666' 2024-02-05T05:42:16.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:16.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:16.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=549755813893 2024-02-05T05:42:16.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 549755813893 2024-02-05T05:42:16.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617666 1-549755813893' 2024-02-05T05:42:16.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:16.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-575525617666 2024-02-05T05:42:16.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:16.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:16.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-575525617666 2024-02-05T05:42:16.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:16.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=575525617666 2024-02-05T05:42:16.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 575525617666' 2024-02-05T05:42:16.463 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 575525617666 2024-02-05T05:42:16.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:16.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 575525617666 2024-02-05T05:42:16.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:17.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:42:17.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:17.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617666 -lt 575525617666 2024-02-05T05:42:17.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:17.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-549755813893 2024-02-05T05:42:17.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:17.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:17.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-549755813893 2024-02-05T05:42:17.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=549755813893 2024-02-05T05:42:17.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 549755813893' 2024-02-05T05:42:17.999 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 549755813893 2024-02-05T05:42:17.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:18.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 549755813893 -lt 549755813893 2024-02-05T05:42:18.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:18.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:18.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:18.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:42:18.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:18.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:18.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:18.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:18.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:18.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:18.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:18.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:42:18.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:18.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:18.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:19.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:42:19.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:19.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:19.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-02-05T05:42:19.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:19.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:19.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:19.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:19.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:19.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-02-05T05:42:19.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:19.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:19.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:42:19.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:19.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:42:19.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:21.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:21.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:21.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:42:21.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:42:21.028 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:42:21.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:21.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:42:21.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:42:21.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:21.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:21.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:21.055 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:21.055+0000 7f65a240e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:21.058 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:21.057+0000 7f65a240e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:21.059+0000 7f65a240e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:21.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:42:21.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:21.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:42:21.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:21.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:21.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:21.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:21.314 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:42:21.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:21.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:21.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:21.591 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:21.591+0000 7f65a240e4c0 -1 Falling back to public interface 2024-02-05T05:42:22.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:22.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:22.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:22.578 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:42:22.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:22.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:22.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:23.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:23.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:23.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:23.853 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:42:23.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:23.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:23.914 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:23.914+0000 7f65a240e4c0 -1 osd.1 135 log_to_monitors true 2024-02-05T05:42:24.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:25.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:25.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:25.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:25.203 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:42:25.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:25.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:25.503 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 139 up_thru 139 down_at 136 last_clean_interval [128,135) [v2:127.0.0.1:6810/1827817329,v1:127.0.0.1:6811/1827817329] [v2:127.0.0.1:6812/1827817329,v1:127.0.0.1:6813/1827817329] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:25.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:25.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:25.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:25.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:25.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:25.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:25.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:25.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:25.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:25.857 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:42:25.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:25.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:25.857 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:25.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=575525617669 2024-02-05T05:42:25.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 575525617669 2024-02-05T05:42:25.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617669' 2024-02-05T05:42:25.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:25.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:26.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=597000454146 2024-02-05T05:42:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 597000454146 2024-02-05T05:42:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617669 1-597000454146' 2024-02-05T05:42:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:26.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-575525617669 2024-02-05T05:42:26.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:26.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:26.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-575525617669 2024-02-05T05:42:26.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:26.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=575525617669 2024-02-05T05:42:26.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 575525617669' 2024-02-05T05:42:26.066 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 575525617669 2024-02-05T05:42:26.066 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:26.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617668 -lt 575525617669 2024-02-05T05:42:26.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:27.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:42:27.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:27.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617669 -lt 575525617669 2024-02-05T05:42:27.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:27.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-597000454146 2024-02-05T05:42:27.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:27.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:27.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-597000454146 2024-02-05T05:42:27.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:27.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=597000454146 2024-02-05T05:42:27.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 597000454146' 2024-02-05T05:42:27.612 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 597000454146 2024-02-05T05:42:27.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:27.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 597000454146 -lt 597000454146 2024-02-05T05:42:27.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:27.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:27.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:28.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:42:28.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:28.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:28.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:28.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:28.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:28.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:28.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:28.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:42:28.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:28.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:28.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:28.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:42:28.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:28.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:28.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-02-05T05:42:28.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:28.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-02-05T05:42:28.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:28.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:42:28.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:28.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:42:28.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:28.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:28.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:28.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:28.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:29.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:29.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-02-05T05:42:29.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:29.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:29.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:42:29.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:29.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:42:29.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-02-05T05:42:30.132 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T05:42:30.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:42:30.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:30.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:30.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:42:30.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:30.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:42:30.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:30.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:42:30.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:42:30.655 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:42:30.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:30.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:42:30.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:42:30.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:30.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:30.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:30.680 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:30.681+0000 7fc760e1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:30.683 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:30.683+0000 7fc760e1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:30.684 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:30.684+0000 7fc760e1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:42:30.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:30.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:31.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:31.203 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:31.204+0000 7fc760e1a4c0 -1 Falling back to public interface 2024-02-05T05:42:32.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:32.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:32.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:32.198 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:42:32.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:32.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:32.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:33.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:33.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:33.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:33.473 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:42:33.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:33.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:33.547 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:33.547+0000 7fc760e1a4c0 -1 osd.1 140 log_to_monitors true 2024-02-05T05:42:33.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:34.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:34.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:34.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:34.755 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:42:34.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:34.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:35.031 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [139,140) [v2:127.0.0.1:6810/1540453248,v1:127.0.0.1:6811/1540453248] [v2:127.0.0.1:6812/1540453248,v1:127.0.0.1:6813/1540453248] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:42:35.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:35.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:35.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:35.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:35.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:35.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:35.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:35.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:35.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:35.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:35.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:35.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:35.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:35.389 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:42:35.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:35.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:35.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:35.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=575525617672 2024-02-05T05:42:35.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 575525617672 2024-02-05T05:42:35.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617672' 2024-02-05T05:42:35.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:35.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290626 2024-02-05T05:42:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290626 2024-02-05T05:42:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-575525617672 1-618475290626' 2024-02-05T05:42:35.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:35.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-575525617672 2024-02-05T05:42:35.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:35.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:35.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-575525617672 2024-02-05T05:42:35.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=575525617672 2024-02-05T05:42:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 575525617672' 2024-02-05T05:42:35.630 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 575525617672 2024-02-05T05:42:35.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:35.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617671 -lt 575525617672 2024-02-05T05:42:35.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:36.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:42:36.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:37.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617671 -lt 575525617672 2024-02-05T05:42:37.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:38.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:42:38.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:38.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 575525617672 -lt 575525617672 2024-02-05T05:42:38.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:38.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290626 2024-02-05T05:42:38.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:38.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:38.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290626 2024-02-05T05:42:38.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:38.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290626 2024-02-05T05:42:38.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2024-02-05T05:42:38.496 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 618475290626 2024-02-05T05:42:38.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:38.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290626 -lt 618475290626 2024-02-05T05:42:38.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:38.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:38.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:39.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:39.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:42:39.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:39.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:39.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:39.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:42:39.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:39.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:39.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:39.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-02-05T05:42:39.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-02-05T05:42:39.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:42:39.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:39.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-02-05T05:42:39.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:39.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:39.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:42:39.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:40.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:40.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-02-05T05:42:40.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:40.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:40.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:42:40.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:40.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:42:40.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-02-05T05:42:41.116 INFO:tasks.workunit.client.0.smithi188.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:41.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:41.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:42:41.889 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:42:41.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:41.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:42:41.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:42:41.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:41.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:41.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:41.916 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:41.917+0000 7fad38e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:41.918 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:41.919+0000 7fad38e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:41.919 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:41.920+0000 7fad38e2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:42.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:42:42.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:42.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:42:42.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:42.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:42.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:42.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:42.169 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:42:42.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:42.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:42.437 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:42.437+0000 7fad38e2a4c0 -1 Falling back to public interface 2024-02-05T05:42:42.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:43.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:43.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:43.458 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:42:43.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:43.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:43.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:44.729 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:44.729+0000 7fad38e2a4c0 -1 osd.0 145 log_to_monitors true 2024-02-05T05:42:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:42:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:45.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:46.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:46.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:46.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:46.004 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:42:46.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:46.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:42:46.275 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [134,145) [v2:127.0.0.1:6802/461940522,v1:127.0.0.1:6803/461940522] [v2:127.0.0.1:6804/461940522,v1:127.0.0.1:6805/461940522] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:42:46.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:46.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:46.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:46.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:46.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:46.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:46.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:46.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:46.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:46.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:46.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:46.632 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:42:46.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:46.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:46.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:46.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127106 2024-02-05T05:42:46.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127106 2024-02-05T05:42:46.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106' 2024-02-05T05:42:46.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:46.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:46.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=618475290629 2024-02-05T05:42:46.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 618475290629 2024-02-05T05:42:46.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127106 1-618475290629' 2024-02-05T05:42:46.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:46.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127106 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127106 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127106 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 639950127106 2024-02-05T05:42:46.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:47.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 639950127106 2024-02-05T05:42:47.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:48.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:42:48.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:48.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127106 -lt 639950127106 2024-02-05T05:42:48.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:48.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-618475290629 2024-02-05T05:42:48.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:48.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:48.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-618475290629 2024-02-05T05:42:48.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:48.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=618475290629 2024-02-05T05:42:48.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 618475290629' 2024-02-05T05:42:48.388 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 618475290629 2024-02-05T05:42:48.388 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:48.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 618475290629 -lt 618475290629 2024-02-05T05:42:48.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:48.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:48.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:48.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:42:48.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:48.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:48.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:48.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:48.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:48.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:48.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:49.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:42:49.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:49.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:49.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:42:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-02-05T05:42:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T05:42:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:49.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:49.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:49.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T05:42:49.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:49.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:49.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:42:49.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:49.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:42:49.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-02-05T05:42:50.977 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T05:42:50.977 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:42:51.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:42:51.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:42:51.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:42:51.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:42:51.497 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:42:51.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:42:51.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:42:51.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:42:51.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:42:51.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:42:51.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:42:51.523 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:51.524+0000 7f5b78d704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:51.525 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:51.526+0000 7f5b78d704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:51.526 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:51.528+0000 7f5b78d704c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:42:51.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:42:51.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:42:51.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:42:51.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:42:51.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:42:51.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:51.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:42:51.771 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:42:51.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:51.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:52.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:52.046 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:52.047+0000 7f5b78d704c0 -1 Falling back to public interface 2024-02-05T05:42:53.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:53.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:53.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:42:53.036 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:42:53.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:53.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:53.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:54.335 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:42:54.336+0000 7f5b78d704c0 -1 osd.1 150 log_to_monitors true 2024-02-05T05:42:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:54.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:42:54.337 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:42:54.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:54.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:54.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:42:55.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:42:55.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:42:55.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:42:55.624 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:42:55.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:42:55.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/941351536,v1:127.0.0.1:6811/941351536] [v2:127.0.0.1:6812/941351536,v1:127.0.0.1:6813/941351536] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:42:56.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:42:56.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:42:56.029 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:42:56.029 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:42:56.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:42:56.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:42:56.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:42:56.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:42:56.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:42:56.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:42:56.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:42:56.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:42:56.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:42:56.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:42:56.383 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:42:56.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:42:56.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:56.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:42:56.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=639950127109 2024-02-05T05:42:56.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 639950127109 2024-02-05T05:42:56.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109' 2024-02-05T05:42:56.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:42:56.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:42:56.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963586 2024-02-05T05:42:56.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963586 2024-02-05T05:42:56.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-639950127109 1-661424963586' 2024-02-05T05:42:56.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:56.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-639950127109 2024-02-05T05:42:56.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:56.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:42:56.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-639950127109 2024-02-05T05:42:56.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:56.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=639950127109 2024-02-05T05:42:56.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 639950127109' 2024-02-05T05:42:56.592 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 639950127109 2024-02-05T05:42:56.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:56.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127108 -lt 639950127109 2024-02-05T05:42:56.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:42:57.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:42:57.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:42:58.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 639950127109 -lt 639950127109 2024-02-05T05:42:58.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:42:58.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963586 2024-02-05T05:42:58.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:42:58.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:42:58.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963586 2024-02-05T05:42:58.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:42:58.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963586 2024-02-05T05:42:58.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2024-02-05T05:42:58.139 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 661424963586 2024-02-05T05:42:58.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:42:58.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963586 -lt 661424963586 2024-02-05T05:42:58.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:42:58.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:58.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:58.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:42:58.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:42:58.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:42:58.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:42:58.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:42:58.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:42:58.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:42:58.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:42:59.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:42:59.071 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:42:59.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:42:59.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:42:59.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:42:59.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:42:59.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:42:59.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:42:59.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-02-05T05:42:59.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:42:59.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:42:59.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:42:59.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:42:59.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:42:59.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:42:59.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:42:59.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:42:59.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:42:59.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:42:59.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:43:00.792 INFO:tasks.workunit.client.0.smithi188.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:01.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:01.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:43:01.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:43:01.396 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:43:01.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:01.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:43:01.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:43:01.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:01.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:01.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:01.423 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:01.424+0000 7f21750184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:01.425 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:01.426+0000 7f21750184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:01.426 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:01.427+0000 7f21750184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:01.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:01.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:01.943 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:01.945+0000 7f21750184c0 -1 Falling back to public interface 2024-02-05T05:43:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:43:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:03.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:04.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:04.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:04.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:04.213 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:43:04.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:04.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:04.216 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:04.217+0000 7f21750184c0 -1 osd.0 156 log_to_monitors true 2024-02-05T05:43:04.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:05.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:05.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:05.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:43:05.502 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:43:05.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:05.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:05.877 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/1648620696,v1:127.0.0.1:6803/1648620696] [v2:127.0.0.1:6804/1648620696,v1:127.0.0.1:6805/1648620696] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:43:05.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:05.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:05.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:05.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:05.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:05.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:05.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:05.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:06.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:06.275 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:43:06.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:06.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:06.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:06.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767362 2024-02-05T05:43:06.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767362 2024-02-05T05:43:06.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362' 2024-02-05T05:43:06.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:06.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:06.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=661424963589 2024-02-05T05:43:06.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 661424963589 2024-02-05T05:43:06.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767362 1-661424963589' 2024-02-05T05:43:06.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:06.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767362 2024-02-05T05:43:06.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:06.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:06.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767362 2024-02-05T05:43:06.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:06.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767362 2024-02-05T05:43:06.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2024-02-05T05:43:06.484 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 687194767362 2024-02-05T05:43:06.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:06.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 687194767362 2024-02-05T05:43:06.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:43:07.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:43:07.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:08.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767362 -lt 687194767362 2024-02-05T05:43:08.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:08.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-661424963589 2024-02-05T05:43:08.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:08.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:08.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-661424963589 2024-02-05T05:43:08.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:08.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=661424963589 2024-02-05T05:43:08.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 661424963589' 2024-02-05T05:43:08.036 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 661424963589 2024-02-05T05:43:08.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:08.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 661424963589 -lt 661424963589 2024-02-05T05:43:08.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:08.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:08.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:08.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:43:08.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:08.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:08.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:08.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:08.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:08.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:08.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:08.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:43:08.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:08.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:08.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:09.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:43:09.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:09.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:09.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:09.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:09.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:09.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:43:09.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-02-05T05:43:10.208 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T05:43:10.209 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T05:43:10.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:43:10.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:10.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:10.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:43:10.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:10.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:10.764 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:43:10.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:43:10.766 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:43:10.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:10.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:43:10.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:43:10.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:10.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:10.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:10.791 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:10.793+0000 7fcec14124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:10.793 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:10.795+0000 7fcec14124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:10.795 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:10.796+0000 7fcec14124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:11.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:43:11.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:11.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:43:11.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:11.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:11.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:11.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:11.043 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:43:11.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:11.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:11.314 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:11.316+0000 7fcec14124c0 -1 Falling back to public interface 2024-02-05T05:43:11.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:12.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:12.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:12.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:12.340 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:43:12.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:12.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:12.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:13.560 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:13.562+0000 7fcec14124c0 -1 osd.1 161 log_to_monitors true 2024-02-05T05:43:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:43:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:13.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:13.919 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 165 up_thru 160 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/386591129,v1:127.0.0.1:6811/386591129] [v2:127.0.0.1:6812/386591129,v1:127.0.0.1:6813/386591129] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:43:13.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:13.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:13.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:14.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:14.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:14.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:14.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:14.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:14.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:14.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:14.477 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:43:14.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:14.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:14.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:14.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=687194767365 2024-02-05T05:43:14.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 687194767365 2024-02-05T05:43:14.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365' 2024-02-05T05:43:14.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:14.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:14.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603842 2024-02-05T05:43:14.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603842 2024-02-05T05:43:14.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-687194767365 1-708669603842' 2024-02-05T05:43:14.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:14.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-687194767365 2024-02-05T05:43:14.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:14.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:14.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-687194767365 2024-02-05T05:43:14.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:14.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=687194767365 2024-02-05T05:43:14.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 687194767365' 2024-02-05T05:43:14.822 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 687194767365 2024-02-05T05:43:14.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767363 -lt 687194767365 2024-02-05T05:43:15.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:43:16.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:43:16.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:16.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 687194767365 -lt 687194767365 2024-02-05T05:43:16.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:16.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603842 2024-02-05T05:43:16.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:16.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:16.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603842 2024-02-05T05:43:16.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:16.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603842 2024-02-05T05:43:16.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2024-02-05T05:43:16.387 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 708669603842 2024-02-05T05:43:16.387 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:16.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603842 -lt 708669603842 2024-02-05T05:43:16.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:16.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:16.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:16.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:17.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:43:17.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:17.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:17.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:17.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:43:17.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:43:17.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:43:17.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:43:17.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:43:17.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:43:17.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:43:17.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:17.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:17.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:17.717 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:17.718 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:17.718 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:17.718 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:17.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:43:17.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:17.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:17.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:43:18.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:43:18.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:43:18.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:43:18.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:43:18.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:43:18.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:43:18.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:43:18.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:43:18.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:18.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:18.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:18.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:18.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:18.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:18.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:43:19.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:19.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:19.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:19.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:43:19.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:43:19.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:43:19.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:43:19.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:43:19.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:43:19.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:43:19.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:43:19.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:43:19.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:43:19.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:43:19.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:43:19.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:43:19.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:43:20.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:43:20.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:20.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:20.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:20.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:20.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:20.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:20.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:20.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:43:20.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:20.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:20.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:43:20.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:43:21.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:43:21.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:43:21.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-02-05T05:43:21.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:43:21.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-05T05:43:21.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:43:21.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:21.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:21.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:21.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:21.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:21.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:21.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:22.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:43:22.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:22.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:22.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:22.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:43:22.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:22.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:22.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:43:22.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-02-05T05:43:22.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-02-05T05:43:22.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-02-05T05:43:22.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:43:22.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:22.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:22.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:22.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:22.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:22.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:22.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:22.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:22.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:22.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:43:22.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:22.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:43:22.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:24.009 INFO:tasks.workunit.client.0.smithi188.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T05:43:24.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:43:24.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:24.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:24.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:43:24.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:24.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:24.606 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:24.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:43:24.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:43:24.608 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:43:24.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:24.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:43:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:43:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:24.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:24.635 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:24.637+0000 7f71c63ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:24.637 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:24.639+0000 7f71c63ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:24.638 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:24.641+0000 7f71c63ad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:24.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:43:24.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:24.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:25.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:25.156 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:25.159+0000 7f71c63ad4c0 -1 Falling back to public interface 2024-02-05T05:43:26.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:26.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:26.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:26.153 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:43:26.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:26.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:26.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:27.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:27.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:27.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:27.430 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:43:27.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:27.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:27.592 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:27.594+0000 7f71c63ad4c0 -1 osd.0 167 log_to_monitors true 2024-02-05T05:43:27.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:28.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:43:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:43:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:28.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/339822511,v1:127.0.0.1:6803/339822511] [v2:127.0.0.1:6804/339822511,v1:127.0.0.1:6805/339822511] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:29.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:29.115 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:29.115 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:29.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:29.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:29.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:29.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:29.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:29.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:29.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:29.491 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:43:29.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:29.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:29.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407618 2024-02-05T05:43:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407618 2024-02-05T05:43:29.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618' 2024-02-05T05:43:29.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:29.593 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:29.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=708669603846 2024-02-05T05:43:29.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 708669603846 2024-02-05T05:43:29.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407618 1-708669603846' 2024-02-05T05:43:29.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:29.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407618 2024-02-05T05:43:29.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:29.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:29.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407618 2024-02-05T05:43:29.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:29.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407618 2024-02-05T05:43:29.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2024-02-05T05:43:29.699 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 734439407618 2024-02-05T05:43:29.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:29.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 734439407618 2024-02-05T05:43:29.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:43:30.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:43:30.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:31.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407618 -lt 734439407618 2024-02-05T05:43:31.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:31.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-708669603846 2024-02-05T05:43:31.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:31.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:31.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-708669603846 2024-02-05T05:43:31.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:31.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=708669603846 2024-02-05T05:43:31.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 708669603846' 2024-02-05T05:43:31.269 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 708669603846 2024-02-05T05:43:31.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:31.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 708669603846 -lt 708669603846 2024-02-05T05:43:31.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:31.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:31.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:31.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:32.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:43:32.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:32.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:32.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:32.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:43:32.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:32.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:32.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:32.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:32.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:43:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:43:32.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-02-05T05:43:33.724 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T05:43:33.724 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T05:43:34.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:34.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:34.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:34.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:34.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:34.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:34.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:34.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:34.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:34.294 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:34.297+0000 7f929bc104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:34.297 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:34.299+0000 7f929bc104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:34.298 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:34.301+0000 7f929bc104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:34.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:34.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:34.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:34.815 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:34.818+0000 7f929bc104c0 -1 Falling back to public interface 2024-02-05T05:43:35.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:35.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:35.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:35.812 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:43:35.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:35.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:36.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:36.948 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:36.950+0000 7f929bc104c0 -1 osd.1 173 log_to_monitors true 2024-02-05T05:43:37.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:37.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:37.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:37.084 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:43:37.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:37.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 177 up_thru 171 down_at 174 last_clean_interval [165,173) [v2:127.0.0.1:6810/2341812463,v1:127.0.0.1:6811/2341812463] [v2:127.0.0.1:6812/2341812463,v1:127.0.0.1:6813/2341812463] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:43:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:37.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:37.388 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:37.388 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:37.389 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:37.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:37.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:37.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:37.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:37.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:37.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:37.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:37.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:37.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:37.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:37.763 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:43:37.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:37.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:37.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:37.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407621 2024-02-05T05:43:37.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407621 2024-02-05T05:43:37.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621' 2024-02-05T05:43:37.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:37.867 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:38.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=760209211394 2024-02-05T05:43:38.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 760209211394 2024-02-05T05:43:38.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407621 1-760209211394' 2024-02-05T05:43:38.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:38.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407621 2024-02-05T05:43:38.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:38.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:38.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407621 2024-02-05T05:43:38.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:38.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407621 2024-02-05T05:43:38.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407621' 2024-02-05T05:43:38.093 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 734439407621 2024-02-05T05:43:38.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:38.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407619 -lt 734439407621 2024-02-05T05:43:38.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:43:39.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:43:39.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:39.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407621 -lt 734439407621 2024-02-05T05:43:39.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:39.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-760209211394 2024-02-05T05:43:39.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:39.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:39.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-760209211394 2024-02-05T05:43:39.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:39.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=760209211394 2024-02-05T05:43:39.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 760209211394' 2024-02-05T05:43:39.707 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 760209211394 2024-02-05T05:43:39.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:39.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 760209211394 -lt 760209211394 2024-02-05T05:43:39.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:39.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:39.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:40.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:40.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:43:40.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:40.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:40.572 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:40.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:43:40.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:43:40.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:43:40.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:43:40.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:43:40.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:43:41.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:43:41.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:41.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:41.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:41.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:41.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:41.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:41.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:41.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:43:41.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:41.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:41.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:43:41.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:43:41.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:43:41.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:43:41.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:43:41.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:43:41.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:43:41.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:43:42.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:43:42.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:42.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:42.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:42.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:42.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:42.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:42.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:42.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:43:42.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:42.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:42.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:42.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:43:42.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:43:43.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:43:43.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:43:43.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:43:43.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:43:43.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:43:43.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:43:43.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:43.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:43.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:43.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:43.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:43.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:43.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:43.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:43:43.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:43.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:43.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:44.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:43:44.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:44.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:44.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-02-05T05:43:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:43:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-02-05T05:43:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:44.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:43:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:43:44.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:44.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:44.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:43:44.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-02-05T05:43:45.170 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T05:43:45.170 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:43:45.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:45.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:43:45.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:43:45.694 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:43:45.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:43:45.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:43:45.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:43:45.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:43:45.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:43:45.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:43:45.720 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:45.722+0000 7ff0b38884c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:45.722 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:45.725+0000 7ff0b38884c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:45.723 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:45.726+0000 7ff0b38884c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:43:45.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:43:45.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:43:45.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:43:45.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:43:45.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:43:45.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:45.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:43:45.969 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:43:45.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:45.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:46.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:46.251 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:46.254+0000 7ff0b38884c0 -1 Falling back to public interface 2024-02-05T05:43:47.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:47.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:47.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:43:47.234 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:43:47.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:47.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:47.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:43:48.379 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:43:48.382+0000 7ff0b38884c0 -1 osd.1 179 log_to_monitors true 2024-02-05T05:43:48.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:43:48.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:43:48.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:43:48.527 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:43:48.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:43:48.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:43:48.881 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 183 up_thru 177 down_at 180 last_clean_interval [177,179) [v2:127.0.0.1:6810/2049382745,v1:127.0.0.1:6811/2049382745] [v2:127.0.0.1:6812/2049382745,v1:127.0.0.1:6813/2049382745] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:43:48.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:43:48.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:43:48.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:43:48.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:43:48.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:43:48.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:43:48.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:43:49.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:43:49.308 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:43:49.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:43:49.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:49.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:43:49.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=734439407624 2024-02-05T05:43:49.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 734439407624 2024-02-05T05:43:49.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624' 2024-02-05T05:43:49.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:43:49.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:43:49.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015170 2024-02-05T05:43:49.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015170 2024-02-05T05:43:49.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-734439407624 1-785979015170' 2024-02-05T05:43:49.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:49.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-734439407624 2024-02-05T05:43:49.586 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:49.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:43:49.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-734439407624 2024-02-05T05:43:49.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:49.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=734439407624 2024-02-05T05:43:49.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2024-02-05T05:43:49.588 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 734439407624 2024-02-05T05:43:49.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:49.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407623 -lt 734439407624 2024-02-05T05:43:49.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:43:50.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:43:50.856 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:51.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407623 -lt 734439407624 2024-02-05T05:43:51.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:43:52.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:43:52.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:43:52.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 734439407624 -lt 734439407624 2024-02-05T05:43:52.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:43:52.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015170 2024-02-05T05:43:52.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:43:52.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:43:52.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015170 2024-02-05T05:43:52.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:43:52.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015170 2024-02-05T05:43:52.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015170' 2024-02-05T05:43:52.412 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 785979015170 2024-02-05T05:43:52.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:43:52.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015170 -lt 785979015170 2024-02-05T05:43:52.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:43:52.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:52.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:53.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:43:53.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:43:53.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:43:53.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:43:53.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:43:53.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:43:53.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:43:53.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:43:53.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:43:53.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:43:53.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:43:53.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:43:53.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:43:53.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:43:53.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:43:53.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-02-05T05:43:53.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-02-05T05:43:53.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-02-05T05:43:53.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-02-05T05:43:53.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-02-05T05:43:53.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-02-05T05:43:53.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-02-05T05:43:53.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-02-05T05:43:53.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-02-05T05:43:53.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:43:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-02-05T05:43:53.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-02-05T05:43:54.010 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: {} 2024-02-05T05:43:54.010 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-02-05T05:43:54.027 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: {} 2024-02-05T05:43:54.027 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:43:54.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T05:43:54.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:43:54.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-02-05T05:43:54.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:43:54.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-02-05T05:43:54.039 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-02-05T05:43:54.039 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:43:54.379 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:43:54.379 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:43:54.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:43:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:43:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:43:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T05:43:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:43:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:43:54.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T05:43:54.382 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:43:54.383 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:43:54.383 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:54.383 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:54.383 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:54.383 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:43:54.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:43:54.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:43:54.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:43:54.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-02-05T05:43:54.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-02-05T05:43:54.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:43:54.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:43:54.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T05:43:54.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-02-05T05:43:54.561 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:43:54.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:43:54.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T05:43:54.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:54.833 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:43:54.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:43:54.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:43:54.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:43:54.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-02-05T05:43:55.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-02-05T05:43:55.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:43:55.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:43:55.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T05:43:55.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T05:43:55.008 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:43:55.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:43:55.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:43:55.326 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:55.327 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:55.327 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:55.327 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:43:55.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:43:55.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:43:55.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:43:55.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:43:55.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T05:43:55.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:43:55.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:43:55.413 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:55.413 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:55.414 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:55.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:43:55.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:43:55.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T05:43:55.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T05:43:55.504 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:43:55.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:43:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:43:55.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:43:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:43:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:43:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T05:43:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:43:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:43:55.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:43:55.771 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:43:55.772 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:55.772 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:55.772 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:55.772 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:43:55.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:43:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:43:55.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:43:55.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:43:55.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-02-05T05:43:55.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-02-05T05:43:55.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T05:43:55.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T05:43:55.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:43:55.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:55.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:55.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:43:56.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T05:40:01.806163+0000 2024-02-05T05:43:56.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T05:43:56.472 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0 on osd.1 to scrub 2024-02-05T05:43:56.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T05:40:01.806163+0000 2024-02-05T05:43:56.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:43:56.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:40:01.806163+0000 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:56.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:43:56.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:40:01.806163+0000 '>' 2024-02-05T05:40:01.806163+0000 2024-02-05T05:43:56.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:43:57.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:43:57.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:57.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:43:57.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:43:57.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:57.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:57.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:43:58.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:40:01.806163+0000 '>' 2024-02-05T05:40:01.806163+0000 2024-02-05T05:43:58.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:43:59.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:43:59.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:43:59.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:43:59.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:43:59.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:43:59.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:43:59.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:43:59.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:43:57.469104+0000 '>' 2024-02-05T05:40:01.806163+0000 2024-02-05T05:43:59.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:43:59.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-02-05T05:43:59.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-02-05T05:43:59.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-02-05T05:43:59.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:43:59.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-02-05T05:43:59.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-02-05T05:43:59.543 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-02-05T05:43:59.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-02-05T05:43:59.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T05:43:59.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-02-05T05:43:59.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-02-05T05:43:59.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=183 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:43:59.576 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:43:59.577 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:43:59.577 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:43:59.577 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:43:59.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T05:43:59.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:43:59.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:43:59.600 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T05:43:59.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T05:43:59.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:43:59.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:43:59.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:43:59.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-02-05T05:43:59.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-02-05T05:43:59.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-02-05T05:43:59.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-02-05T05:43:59.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:43:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:43:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:43:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:43:59.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:43:59.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:43:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:43:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:43:59.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:44:00.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-02-05T05:44:00.842 INFO:tasks.workunit.client.0.smithi188.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:44:01.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:44:01.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:44:01.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:44:01.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:44:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:01.398+0000 7f538a8d84c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:01.397 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:01.400+0000 7f538a8d84c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:01.398 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:01.402+0000 7f538a8d84c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:01.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:44:01.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:44:01.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:44:01.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:44:01.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:44:01.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:01.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:44:01.645 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:44:01.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:01.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:01.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:01.921 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:01.925+0000 7f538a8d84c0 -1 Falling back to public interface 2024-02-05T05:44:02.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:02.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:02.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:44:02.913 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:44:02.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:02.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:03.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:04.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:04.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:04.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:44:04.193 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:44:04.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:04.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:04.263 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:04.266+0000 7f538a8d84c0 -1 osd.0 185 log_to_monitors true 2024-02-05T05:44:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:05.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:05.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:05.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:44:05.492 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:44:05.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:05.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:05.767 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 189 up_thru 189 down_at 186 last_clean_interval [171,185) [v2:127.0.0.1:6802/3977601696,v1:127.0.0.1:6803/3977601696] [v2:127.0.0.1:6804/3977601696,v1:127.0.0.1:6805/3977601696] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:44:05.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:44:05.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:44:05.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:44:05.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:44:05.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:44:05.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:44:05.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:44:05.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:44:05.856 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:44:06.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:44:06.119 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:44:06.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:44:06.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:06.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:44:06.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818946 2024-02-05T05:44:06.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818946 2024-02-05T05:44:06.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946' 2024-02-05T05:44:06.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:06.222 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:44:06.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=785979015174 2024-02-05T05:44:06.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 785979015174 2024-02-05T05:44:06.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818946 1-785979015174' 2024-02-05T05:44:06.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:06.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818946 2024-02-05T05:44:06.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:06.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:44:06.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818946 2024-02-05T05:44:06.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:06.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818946 2024-02-05T05:44:06.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818946' 2024-02-05T05:44:06.327 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 811748818946 2024-02-05T05:44:06.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:06.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 811748818946 2024-02-05T05:44:06.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:44:07.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:44:07.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:07.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818946 -lt 811748818946 2024-02-05T05:44:07.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:07.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-785979015174 2024-02-05T05:44:07.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:07.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:44:07.867 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-785979015174 2024-02-05T05:44:07.867 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:07.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=785979015174 2024-02-05T05:44:07.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 785979015174' 2024-02-05T05:44:07.869 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 785979015174 2024-02-05T05:44:07.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:44:08.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 785979015174 -lt 785979015174 2024-02-05T05:44:08.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:44:08.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:08.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:08.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:44:08.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:08.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:08.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:08.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:08.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:08.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:08.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:08.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:44:08.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:08.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:08.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:09.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:44:09.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:44:09.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:44:09.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:44:09.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:44:09.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:44:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:44:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-02-05T05:44:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:44:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:44:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:44:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:44:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:44:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-02-05T05:44:10.032 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T05:44:10.032 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T05:44:10.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:44:10.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:44:10.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:44:10.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:44:10.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:44:10.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:44:10.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:44:10.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:44:10.552 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:44:10.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:44:10.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:44:10.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:44:10.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:44:10.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:44:10.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:44:10.577 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:10.581+0000 7fd73709e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:10.579 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:10.583+0000 7fd73709e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:10.580 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:10.584+0000 7fd73709e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:10.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:10.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:11.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:11.100 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:11.104+0000 7fd73709e4c0 -1 Falling back to public interface 2024-02-05T05:44:12.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:12.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:12.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:44:12.090 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:44:12.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:12.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:12.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:13.253 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:13.256+0000 7fd73709e4c0 -1 osd.1 191 log_to_monitors true 2024-02-05T05:44:13.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:13.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:13.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:44:13.366 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:44:13.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:13.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 195 up_thru 189 down_at 192 last_clean_interval [183,191) [v2:127.0.0.1:6810/2321595739,v1:127.0.0.1:6811/2321595739] [v2:127.0.0.1:6812/2321595739,v1:127.0.0.1:6813/2321595739] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:44:13.705 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:44:13.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:44:13.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:44:13.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:44:13.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:44:13.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:44:13.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:44:13.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:44:13.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:44:13.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:44:14.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:44:14.092 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:44:14.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:44:14.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:14.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:44:14.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=811748818948 2024-02-05T05:44:14.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 811748818948 2024-02-05T05:44:14.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818948' 2024-02-05T05:44:14.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:14.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:44:14.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622722 2024-02-05T05:44:14.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622722 2024-02-05T05:44:14.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-811748818948 1-837518622722' 2024-02-05T05:44:14.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:14.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-811748818948 2024-02-05T05:44:14.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:14.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:44:14.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-811748818948 2024-02-05T05:44:14.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:14.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=811748818948 2024-02-05T05:44:14.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 811748818948' 2024-02-05T05:44:14.407 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 811748818948 2024-02-05T05:44:14.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:14.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818947 -lt 811748818948 2024-02-05T05:44:14.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:44:15.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:16.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 811748818949 -lt 811748818948 2024-02-05T05:44:16.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:16.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622722 2024-02-05T05:44:16.213 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:16.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:44:16.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622722 2024-02-05T05:44:16.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:16.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622722 2024-02-05T05:44:16.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622722' 2024-02-05T05:44:16.216 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 837518622722 2024-02-05T05:44:16.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:44:16.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622722 -lt 837518622722 2024-02-05T05:44:16.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:44:16.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:16.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:16.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:44:16.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:16.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:16.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:16.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:16.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:16.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:16.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:17.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:44:17.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:17.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:17.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:17.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:44:17.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:44:17.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:44:17.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:44:17.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:44:17.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:44:17.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:44:17.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:17.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:17.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:17.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:17.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:17.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:17.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:17.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:44:17.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:17.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:17.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:18.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:44:18.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:44:18.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:44:18.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:44:18.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:44:18.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:44:18.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:44:18.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:44:18.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:44:18.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:44:18.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:44:18.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:44:18.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:44:18.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:44:18.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:44:18.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:18.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:18.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:18.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:18.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:18.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:18.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:18.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:44:18.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:18.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:18.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:19.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:44:19.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:44:19.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:44:19.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:44:19.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:44:19.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:44:19.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:44:20.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:44:20.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:20.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:20.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:20.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:20.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:20.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:20.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:20.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:44:20.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:20.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:20.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:20.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:44:20.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-02-05T05:44:20.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:44:20.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:44:20.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:44:20.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:44:20.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:44:20.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:44:20.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:44:20.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:44:20.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:44:20.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-02-05T05:44:20.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:44:20.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-02-05T05:44:21.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:44:21.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:21.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:21.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:21.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:21.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:21.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:21.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:22.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:44:22.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:22.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:22.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:22.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:44:22.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:44:22.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:44:22.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:44:22.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:44:22.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:44:22.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:44:22.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:44:22.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:44:22.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:23.995 INFO:tasks.workunit.client.0.smithi188.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-02-05T05:44:24.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:44:24.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:44:24.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:44:24.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:44:24.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:44:24.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:44:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:44:24.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:44:24.521 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:44:24.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:44:24.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:44:24.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:44:24.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:44:24.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:44:24.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:44:24.548 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:24.552+0000 7fa3d17fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:24.550 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:24.554+0000 7fa3d17fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:24.551 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:24.556+0000 7fa3d17fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:24.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:24.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:25.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:25.068 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:25.073+0000 7fa3d17fb4c0 -1 Falling back to public interface 2024-02-05T05:44:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:44:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:44:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:26.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:27.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:27.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:27.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:44:27.337 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:44:27.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:27.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:27.468 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:27.473+0000 7fa3d17fb4c0 -1 osd.0 198 log_to_monitors true 2024-02-05T05:44:27.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:28.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:28.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:28.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:44:28.617 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:44:28.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:28.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:44:28.890 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 201 up_thru 201 down_at 198 last_clean_interval [189,198) [v2:127.0.0.1:6802/3805495386,v1:127.0.0.1:6803/3805495386] [v2:127.0.0.1:6804/3805495386,v1:127.0.0.1:6805/3805495386] exists,up e7efffd6-654c-4e99-a6be-2f48696bf81a 2024-02-05T05:44:28.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:44:28.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:44:28.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:44:28.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:44:28.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:44:28.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:44:28.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:44:28.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:44:28.979 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:44:29.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:44:29.244 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:44:29.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:44:29.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:29.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:44:29.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426498 2024-02-05T05:44:29.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426498 2024-02-05T05:44:29.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498' 2024-02-05T05:44:29.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:29.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:44:29.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=837518622726 2024-02-05T05:44:29.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 837518622726 2024-02-05T05:44:29.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426498 1-837518622726' 2024-02-05T05:44:29.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:29.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426498 2024-02-05T05:44:29.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:29.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:44:29.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426498 2024-02-05T05:44:29.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:29.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426498 2024-02-05T05:44:29.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426498' 2024-02-05T05:44:29.480 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 863288426498 2024-02-05T05:44:29.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:29.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 863288426498 2024-02-05T05:44:29.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:44:30.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:44:30.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:31.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 863288426498 2024-02-05T05:44:31.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:44:32.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:44:32.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:32.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426498 -lt 863288426498 2024-02-05T05:44:32.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:32.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-837518622726 2024-02-05T05:44:32.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:32.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:44:32.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-837518622726 2024-02-05T05:44:32.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:32.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=837518622726 2024-02-05T05:44:32.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 837518622726' 2024-02-05T05:44:32.305 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 837518622726 2024-02-05T05:44:32.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:44:32.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 837518622726 -lt 837518622726 2024-02-05T05:44:32.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:44:32.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:32.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:32.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:44:32.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:32.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:32.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:32.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:32.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:32.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:32.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:33.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:44:33.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:33.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:33.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:33.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:44:33.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:44:33.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:44:33.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:33.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:44:33.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:44:33.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:44:33.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:44:33.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:44:33.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:33.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:44:33.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:44:33.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:44:33.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:44:33.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:44:33.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-02-05T05:44:34.802 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-02-05T05:44:34.802 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:44:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:44:35.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:44:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:44:35.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:44:35.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:44:35.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:44:35.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:44:35.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:44:35.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:44:35.356 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:35.361+0000 7ffb0060c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:35.358 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:35.363+0000 7ffb0060c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:35.360 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:35.364+0000 7ffb0060c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:44:35.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:44:35.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:44:35.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:44:35.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:44:35.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:44:35.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:35.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:44:35.609 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:44:35.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:35.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:35.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:35.884+0000 7ffb0060c4c0 -1 Falling back to public interface 2024-02-05T05:44:36.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:36.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:36.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:44:36.873 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:44:36.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:36.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:37.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:44:38.006 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:44:38.011+0000 7ffb0060c4c0 -1 osd.1 203 log_to_monitors true 2024-02-05T05:44:38.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:44:38.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:44:38.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:44:38.150 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:44:38.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:44:38.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:44:38.509 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 207 up_thru 201 down_at 204 last_clean_interval [195,203) [v2:127.0.0.1:6810/1225502983,v1:127.0.0.1:6811/1225502983] [v2:127.0.0.1:6812/1225502983,v1:127.0.0.1:6813/1225502983] exists,up 338533ae-6171-4c8a-951a-98411cc8b12b 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:44:38.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:44:38.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:44:38.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:44:38.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:44:38.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:44:38.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:44:38.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:44:38.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:44:38.599 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:44:38.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:44:38.867 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:44:38.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:44:38.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:38.867 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:44:39.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426501 2024-02-05T05:44:39.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426501 2024-02-05T05:44:39.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501' 2024-02-05T05:44:39.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:39.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:44:39.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230274 2024-02-05T05:44:39.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230274 2024-02-05T05:44:39.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426501 1-889058230274' 2024-02-05T05:44:39.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:39.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426501 2024-02-05T05:44:39.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:39.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:44:39.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426501 2024-02-05T05:44:39.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:39.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426501 2024-02-05T05:44:39.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426501' 2024-02-05T05:44:39.282 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 863288426501 2024-02-05T05:44:39.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:39.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426501 -lt 863288426501 2024-02-05T05:44:39.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:39.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230274 2024-02-05T05:44:39.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:39.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:44:39.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230274 2024-02-05T05:44:39.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:39.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230274 2024-02-05T05:44:39.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230274' 2024-02-05T05:44:39.588 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 889058230274 2024-02-05T05:44:39.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:44:39.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 889058230274 2024-02-05T05:44:39.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:44:40.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:44:40.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:44:41.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 889058230274 2024-02-05T05:44:41.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:44:42.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:44:42.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:44:42.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230274 -lt 889058230274 2024-02-05T05:44:42.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:44:42.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:42.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:42.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:44:42.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:42.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:42.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:42.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:42.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:42.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:42.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:43.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T05:44:43.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:43.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:43.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:43.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-02-05T05:44:43.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-02-05T05:44:43.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:44:43.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-02-05T05:44:43.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:44:43.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:44:43.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:44:43.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:43.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:43.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:43.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:43.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:43.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:43.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:43.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:44:43.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:43.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:43.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:44.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:44:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:44:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:44:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:44:44.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:44:44.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T05:44:44.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:44:44.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-02-05T05:44:44.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:44:44.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-02-05T05:44:44.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-02-05T05:44:44.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:44:44.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:44:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T05:44:44.376 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:44:44.376 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:44:44.376 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:44.376 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:44.376 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:44.376 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:44:44.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:44:44.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:44:44.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:44:44.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:44:44.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:44:44.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:44:44.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T05:44:44.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:44:44.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-02-05T05:44:44.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:44:44.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-02-05T05:44:44.553 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-02-05T05:44:44.553 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:44:44.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:44:44.818 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:44:44.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:44:44.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:44:44.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:44:44.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T05:44:44.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:44:44.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:44:44.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:44.821 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:44.822 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:44:44.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:44:44.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:44:44.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:44:44.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:44:44.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T05:44:44.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:44:44.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:44:44.907 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:44.908 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:44.908 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:44.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:44:44.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:44:44.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-02-05T05:44:44.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-02-05T05:44:44.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:44:44.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:44:44.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T05:44:44.997 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T05:44:44.998 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:44:45.289 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:44:45.289 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:44:45.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:44:45.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:44:45.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:44:45.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-02-05T05:44:45.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:44:45.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:44:45.290 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:45.293 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:44:45.293 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:44:45.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:44:45.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:44:45.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:44:45.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-02-05T05:44:45.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T05:44:45.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T05:44:45.380 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:45.381 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:45.381 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:45.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.0.asok 2024-02-05T05:44:45.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:44:45.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-02-05T05:44:45.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-02-05T05:44:45.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-02-05T05:44:45.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-02-05T05:44:45.469 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-02-05T05:44:45.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-02-05T05:44:45.738 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:44:45.739 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:44:45.740 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:45.740 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:45.740 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:45.741 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:44:45.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69197/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-02-05T05:44:45.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-02-05T05:44:45.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:45.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-osd.1.asok 2024-02-05T05:44:45.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-02-05T05:44:45.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.69197/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-02-05T05:44:45.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-02-05T05:44:46.026 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: {} 2024-02-05T05:44:46.026 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0: osd_max_object_size = '' 2024-02-05T05:44:46.043 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: {} 2024-02-05T05:44:46.043 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1: osd_max_object_size = '' 2024-02-05T05:44:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-02-05T05:44:46.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-02-05T05:44:46.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:44:46.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:46.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:44:46.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:46.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:44:46.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T05:40:01.806163+0000 2024-02-05T05:44:46.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-02-05T05:44:46.597 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-02-05T05:44:46.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-02-05T05:40:01.806163+0000 last_deep_scrub_stamp 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:40:01.806163+0000 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:46.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:44:46.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:40:01.806163+0000 '>' 2024-02-05T05:40:01.806163+0000 2024-02-05T05:44:46.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:44:47.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:44:47.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:47.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:44:47.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:47.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:44:47.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:47.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:44:48.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:40:01.806163+0000 '>' 2024-02-05T05:40:01.806163+0000 2024-02-05T05:44:48.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:44:49.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:44:49.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:49.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-02-05T05:44:49.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:49.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:44:49.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:49.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:44:47.219672+0000 '>' 2024-02-05T05:40:01.806163+0000 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-02-05T05:44:49.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-02-05T05:44:49.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-02-05T05:44:49.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-02-05T05:44:49.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-02-05T05:44:49.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-02-05T05:44:49.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-02-05T05:44:49.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T05:44:49.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-02-05T05:44:49.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-02-05T05:44:49.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-02-05T05:44:49.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=207 2024-02-05T05:44:49.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T05:44:49.897 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:44:49.897 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:44:49.897 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:44:49.897 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:44:49.897 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T05:44:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:44:49.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-02-05T05:44:49.921 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T05:44:49.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-02-05T05:44:49.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:44:49.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:44:49.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:44:49.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-02-05T05:44:49.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-02-05T05:44:49.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-02-05T05:44:49.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-02-05T05:44:49.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-02-05T05:44:49.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:49.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:44:49.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:49.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:44:50.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:50.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-02-05T05:44:50.470 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0 on osd.1 to repair 2024-02-05T05:44:50.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:50.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:44:50.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:44:47.219672+0000 '>' 2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:50.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:44:51.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:44:51.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:51.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:44:51.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:51.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:44:51.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:51.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:44:52.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:44:47.219672+0000 '>' 2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:52.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:44:53.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:44:53.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:53.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:44:53.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:53.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:44:53.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:53.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:44:53.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:44:47.219672+0000 '>' 2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:53.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:44:54.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:44:54.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:54.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:44:54.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:54.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:44:54.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:54.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:44:54.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:44:47.219672+0000 '>' 2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:54.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:44:55.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:44:55.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:44:55.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:44:55.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:44:55.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:44:55.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:44:55.596 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:44:51.883330+0000 '>' 2024-02-05T05:44:47.219672+0000 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:44:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:44:55.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:44:55.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:44:55.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:44:55.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:44:55.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:44:55.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:44:55.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:44:55.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:44:55.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:44:56.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:44:56.220 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:44:56.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:44:56.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:56.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:44:56.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=863288426505 2024-02-05T05:44:56.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 863288426505 2024-02-05T05:44:56.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505' 2024-02-05T05:44:56.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:44:56.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:44:56.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=889058230278 2024-02-05T05:44:56.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 889058230278 2024-02-05T05:44:56.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-863288426505 1-889058230278' 2024-02-05T05:44:56.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:56.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-863288426505 2024-02-05T05:44:56.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:56.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:44:56.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-863288426505 2024-02-05T05:44:56.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:56.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=863288426505 2024-02-05T05:44:56.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 863288426505' 2024-02-05T05:44:56.429 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 863288426505 2024-02-05T05:44:56.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:56.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426504 -lt 863288426505 2024-02-05T05:44:56.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:44:57.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:44:57.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:44:57.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 863288426505 -lt 863288426505 2024-02-05T05:44:57.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:44:57.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-889058230278 2024-02-05T05:44:57.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:44:57.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:44:57.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-889058230278 2024-02-05T05:44:57.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:44:57.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=889058230278 2024-02-05T05:44:57.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 889058230278' 2024-02-05T05:44:57.967 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 889058230278 2024-02-05T05:44:57.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:44:58.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 889058230278 -lt 889058230278 2024-02-05T05:44:58.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:44:58.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:58.234 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:44:58.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:44:58.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:44:58.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:44:58.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:44:58.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:44:59.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:44:59.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:44:59.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:44:59.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-02-05T05:44:59.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-02-05T05:44:59.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-02-05T05:44:59.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-02-05T05:44:59.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-02-05T05:44:59.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-02-05T05:44:59.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-02-05T05:44:59.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-02-05T05:44:59.630 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'csr_pool' removed 2024-02-05T05:44:59.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:44:59.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:44:59.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:44:59.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:44:59.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:44:59.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:44:59.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:44:59.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:44:59.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:44:59.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:44:59.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:44:59.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:44:59.752 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:44:59.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:44:59.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:44:59.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:44:59.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:44:59.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:44:59.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:44:59.755 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:44:59.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:44:59.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:44:59.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:44:59.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:44:59.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:59.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:59.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:44:59.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:44:59.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:44:59.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:44:59.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:44:59.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:44:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:44:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:44:59.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:44:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:44:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:44:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:44:59.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:44:59.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:44:59.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:44:59.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:44:59.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:44:59.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:59.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:59.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:44:59.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:44:59.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:44:59.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:44:59.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:44:59.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:44:59.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:44:59.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-02-05T05:44:59.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T05:44:59.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:44:59.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:44:59.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:44:59.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:44:59.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:44:59.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T05:45:00.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:45:00.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:45:00.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:45:00.173 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:45:00.173 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:45:00.173 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:45:00.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:45:00.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T05:45:00.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:45:00.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:45:00.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:45:00.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:45:00.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:45:00.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:45:00.341 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:45:00.341 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:45:00.341 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:45:00.342 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:45:00.342 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:45:00.342 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:45:00.342 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:45:00.342 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:45:00.342 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:45:00.422 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:45:00.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:45:00.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:45:00.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-02-05T05:45:00.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:45:00.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:45:00.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:45:00.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:45:00.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:45:00.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:45:01.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:45:01.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:45:01.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:45:01.300 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:45:01.300 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:45:01.300 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:45:01.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:45:01.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:45:01.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:45:01.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:45:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:45:01.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:45:01.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:45:01.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:45:01.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=038b4cb7-21f1-477e-8dca-d018fa778d2f 2024-02-05T05:45:01.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 038b4cb7-21f1-477e-8dca-d018fa778d2f' 2024-02-05T05:45:01.335 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 038b4cb7-21f1-477e-8dca-d018fa778d2f 2024-02-05T05:45:01.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:45:01.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDddcBl7fkpFRAAQX/a5hXeYP0aaj8HCiEatQ== 2024-02-05T05:45:01.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDddcBl7fkpFRAAQX/a5hXeYP0aaj8HCiEatQ=="}' 2024-02-05T05:45:01.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 038b4cb7-21f1-477e-8dca-d018fa778d2f -i td/osd-scrub-repair/0/new.json 2024-02-05T05:45:01.540 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:45:01.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:45:01.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDddcBl7fkpFRAAQX/a5hXeYP0aaj8HCiEatQ== --osd-uuid 038b4cb7-21f1-477e-8dca-d018fa778d2f 2024-02-05T05:45:01.578 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:01.584+0000 7fa179cbe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:01.580 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:01.586+0000 7fa179cbe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:01.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:01.587+0000 7fa179cbe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:01.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:01.587+0000 7fa179cbe4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:45:01.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:01.587+0000 7fa179cbe4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:45:01.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:01.587+0000 7fa179cbe4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:45:01.649 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:01.654+0000 7fa179cbe4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:45:07.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:45:07.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:45:07.124 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:45:07.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:45:07.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:45:07.466 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:45:07.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:45:07.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:45:07.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:45:07.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:45:07.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:45:07.492 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:07.498+0000 7fd15236c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:07.508 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:07.513+0000 7fd15236c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:07.508 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:07.514+0000 7fd15236c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:07.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:45:07.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:45:07.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:45:07.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:45:07.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:45:07.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:07.770 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:45:07.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:45:07.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:07.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:08.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:08.043 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:08.049+0000 7fd15236c4c0 -1 Falling back to public interface 2024-02-05T05:45:09.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:09.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:09.037 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:45:09.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:45:09.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:09.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:09.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:10.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:10.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:10.312 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:45:10.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:45:10.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:10.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:10.383 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:10.388+0000 7fd15236c4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:45:10.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:11.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:11.592 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:45:11.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:11.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:45:11.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:11.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:11.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:12.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:12.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:12.867 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:45:12.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:45:12.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:12.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:13.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:14.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:14.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:14.145 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:45:14.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:45:14.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:14.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:14.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:15.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:15.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:15.424 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:45:15.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:45:15.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:15.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:15.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:16.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:16.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:16.701 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:45:16.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:45:16.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:16.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:16.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:17.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:17.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:17.979 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:45:17.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:45:17.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:17.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:18.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:19.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:19.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:19.389 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:45:19.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:45:19.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:19.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:19.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:20.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:20.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:20.669 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:45:20.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:45:20.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:20.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:20.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:21.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:21.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:21.948 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:45:21.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:45:21.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:21.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:22.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:23.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:23.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:23.480 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:45:23.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:45:23.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:23.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:24.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:25.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:25.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:25.064 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:45:25.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:45:25.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:25.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:25.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:26.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:26.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:26.762 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:45:26.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:45:26.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:26.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:27.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:28.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:28.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:28.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:45:28.295 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:45:28.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:28.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:28.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:29.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:29.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:29.819 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:45:29.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:45:29.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:29.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:30.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:31.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:31.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:31.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:45:31.513 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:45:31.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:31.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:31.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:32.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:32.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:32.813 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:45:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:45:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:32.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:45:33.084 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1931799207,v1:127.0.0.1:6803/1931799207] [v2:127.0.0.1:6804/1931799207,v1:127.0.0.1:6805/1931799207] exists,up 038b4cb7-21f1-477e-8dca-d018fa778d2f 2024-02-05T05:45:33.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:45:33.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:45:33.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:45:33.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:45:33.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1ff70854-a136-4e5c-b63b-f7e5b0c9a2f2 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1ff70854-a136-4e5c-b63b-f7e5b0c9a2f2' 2024-02-05T05:45:33.091 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 1ff70854-a136-4e5c-b63b-f7e5b0c9a2f2 2024-02-05T05:45:33.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:45:33.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9dcBl1My1BhAAVNW2H6zuIIse96/z6U0uXg== 2024-02-05T05:45:33.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9dcBl1My1BhAAVNW2H6zuIIse96/z6U0uXg=="}' 2024-02-05T05:45:33.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1ff70854-a136-4e5c-b63b-f7e5b0c9a2f2 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:45:33.493 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:45:33.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:45:33.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9dcBl1My1BhAAVNW2H6zuIIse96/z6U0uXg== --osd-uuid 1ff70854-a136-4e5c-b63b-f7e5b0c9a2f2 2024-02-05T05:45:33.533 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:33.540+0000 7fe12500d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:33.586 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:33.593+0000 7fe12500d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:33.594+0000 7fe12500d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:33.594+0000 7fe12500d4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:45:33.587 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:33.594+0000 7fe12500d4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:45:33.588 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:33.594+0000 7fe12500d4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:45:33.661 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:33.668+0000 7fe12500d4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:45:39.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:45:39.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:45:39.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:45:39.342 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:45:39.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:45:39.689 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:45:39.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:45:39.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:45:39.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:45:39.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:45:39.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:45:39.715 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:39.721+0000 7f27bc0124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:39.717 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:39.723+0000 7f27bc0124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:39.718 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:39.725+0000 7f27bc0124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:45:39.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:45:39.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:45:39.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:45:39.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:45:39.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:45:39.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:39.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:45:39.957 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:45:39.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:39.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:40.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:40.247 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:40.254+0000 7f27bc0124c0 -1 Falling back to public interface 2024-02-05T05:45:41.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:41.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:41.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:45:41.228 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:45:41.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:41.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:41.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:42.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:42.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:42.507 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:45:42.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:45:42.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:42.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:42.561 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:45:42.568+0000 7f27bc0124c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:45:42.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:43.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:43.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:43.847 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:45:43.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:45:43.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:43.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:44.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:45.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:45.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:45.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:45:45.171 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:45:45.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:45.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:45.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:45:46.611 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:45:46.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:46.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:46.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:47.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:47.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:47.888 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:45:47.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:45:47.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:47.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:48.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:49.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:49.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:49.167 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:45:49.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:45:49.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:49.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:49.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:50.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:50.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:50.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:45:50.583 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:45:50.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:50.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:50.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:51.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:51.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:51.863 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:45:51.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:45:51.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:51.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:52.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:53.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:53.189 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:45:53.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:53.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:45:53.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:53.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:53.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:54.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:54.564 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:45:54.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:54.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:45:54.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:54.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:55.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:56.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:56.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:56.083 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:45:56.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:45:56.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:56.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:56.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:57.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:57.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:57.633 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:45:57.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:45:57.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:57.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:45:58.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:45:59.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:45:59.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:45:59.255 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:45:59.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:45:59.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:45:59.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:00.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:01.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:01.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:01.041 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:46:01.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:46:01.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:01.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:01.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:02.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:02.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:02.785 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:46:02.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:46:02.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:02.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:03.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:04.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:04.197 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:46:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:46:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:04.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:04.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:05.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:05.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:05.623 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:46:05.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:46:05.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:05.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:05.921 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/307482388,v1:127.0.0.1:6811/307482388] [v2:127.0.0.1:6812/307482388,v1:127.0.0.1:6813/307482388] exists,up 1ff70854-a136-4e5c-b63b-f7e5b0c9a2f2 2024-02-05T05:46:05.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:46:05.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:46:05.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:46:05.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-02-05T05:46:05.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:46:06.179 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:46:06.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:46:06.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:46:06.593 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:46:06.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:46:07.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:46:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-02-05T05:46:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:46:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:46:09.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:46:09.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:46:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:46:09.373 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:46:09.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:46:09.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:46:09.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:46:09.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:46:09.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:46:09.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:46:09.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:46:09.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:46:09.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:46:09.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:46:09.739 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:46:09.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:46:09.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:09.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:46:09.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-02-05T05:46:09.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-02-05T05:46:09.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-02-05T05:46:09.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:09.842 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:46:09.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T05:46:09.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T05:46:09.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672964' 2024-02-05T05:46:09.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:09.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-02-05T05:46:09.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:09.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:46:09.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-02-05T05:46:09.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:09.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-02-05T05:46:09.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-02-05T05:46:09.948 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836490 2024-02-05T05:46:09.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:46:10.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-02-05T05:46:10.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:46:11.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:46:11.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:46:11.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-02-05T05:46:11.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:11.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T05:46:11.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:11.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:46:11.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T05:46:11.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:11.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T05:46:11.503 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672964 2024-02-05T05:46:11.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T05:46:11.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:46:11.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672964 2024-02-05T05:46:11.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:46:11.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:11.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:12.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:46:12.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:46:12.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:46:12.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:46:12.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:46:12.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:46:12.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:46:12.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:46:12.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:46:12.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:46:12.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:12.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:12.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:46:12.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:46:12.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:46:12.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-02-05T05:46:12.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-02-05T05:46:13.118 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'foo' created 2024-02-05T05:46:13.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:46:14.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-02-05T05:46:14.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-02-05T05:46:14.508 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'csr_pool' created 2024-02-05T05:46:14.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:46:15.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-02-05T05:46:15.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:46:15.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:46:15.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:46:15.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:46:15.522 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:46:15.522 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:46:15.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:46:15.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:46:15.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:46:15.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:46:15.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:46:15.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:46:15.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:46:15.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:46:15.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:46:15.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:46:15.893 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:46:15.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:46:15.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:15.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:46:15.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-02-05T05:46:15.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-02-05T05:46:15.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-02-05T05:46:15.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:15.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:46:16.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-02-05T05:46:16.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-02-05T05:46:16.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672967' 2024-02-05T05:46:16.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:16.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-02-05T05:46:16.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:16.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:46:16.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-02-05T05:46:16.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:16.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-02-05T05:46:16.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-02-05T05:46:16.104 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836492 2024-02-05T05:46:16.105 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:46:16.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-02-05T05:46:16.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:46:17.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:46:17.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:46:17.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-02-05T05:46:17.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:17.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-02-05T05:46:17.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:17.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:46:17.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-02-05T05:46:17.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:17.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-02-05T05:46:17.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-02-05T05:46:17.656 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672967 2024-02-05T05:46:17.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:46:17.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-02-05T05:46:17.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:46:17.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:17.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:18.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:46:18.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:46:18.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:46:18.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:46:18.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:46:18.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:46:18.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:46:18.265 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:46:18.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:46:18.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:46:18.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:18.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:18.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T05:46:18.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-02-05T05:46:18.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:46:18.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-02-05T05:46:18.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:46:18.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:46:18.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:46:18.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:46:18.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:46:18.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:46:18.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:46:18.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:46:18.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:46:18.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:46:19.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:46:19.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:46:19.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:19.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:46:19.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:46:19.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:46:19.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:46:19.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:46:19.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:46:19.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:46:20.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:46:20.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:46:20.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:46:20.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:46:20.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:46:20.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:46:20.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:46:20.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:46:20.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:46:20.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:46:20.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:20.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:20.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:46:20.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:46:21.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:46:21.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:46:21.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-02-05T05:46:21.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:46:21.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-02-05T05:46:21.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:46:21.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:46:21.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:46:21.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:46:21.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:46:21.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:46:21.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:46:21.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:46:21.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:46:21.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:46:21.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:21.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:22.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:46:22.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:46:22.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:46:22.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:46:22.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:46:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:46:22.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:46:22.827 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:46:22.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:46:22.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:46:22.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:46:23.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-02-05T05:46:23.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:46:23.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:46:23.673 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:46:23.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:46:24.167 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:46:24.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:46:24.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:46:24.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:46:24.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-02-05T05:46:24.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-02-05T05:46:24.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-02-05T05:46:24.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-02-05T05:46:24.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-02-05T05:46:24.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-02-05T05:46:24.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:46:24.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-02-05T05:46:24.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-02-05T05:46:24.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-02-05T05:46:24.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-02-05T05:46:24.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-02-05T05:46:24.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:46:25.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-02-05T05:46:25.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-02-05T05:46:25.406 INFO:tasks.workunit.client.0.smithi188.stdout:created pool csr_pool snap snap1 2024-02-05T05:46:25.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-02-05T05:46:25.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-02-05T05:46:25.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T05:46:25.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-02-05T05:46:25.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-02-05T05:46:25.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-02-05T05:46:25.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-02-05T05:46:25.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-02-05T05:46:25.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:46:25.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:46:25.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:46:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:46:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-02-05T05:46:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:46:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:46:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:46:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:46:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:46:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-02-05T05:46:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-02-05T05:46:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:46:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:46:27.826 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:46:27.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:46:27.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:46:27.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:46:27.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:46:27.828 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:46:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:46:27.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:46:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:46:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:46:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:46:27.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:46:27.854 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:27.862+0000 7fd15b0af4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:27.856 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:27.864+0000 7fd15b0af4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:27.857 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:27.866+0000 7fd15b0af4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:28.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:28.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:28.375 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:28.384+0000 7fd15b0af4c0 -1 Falling back to public interface 2024-02-05T05:46:29.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:29.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:29.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:46:29.371 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:46:29.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:29.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:29.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:30.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:30.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:30.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:46:30.664 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:46:30.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:30.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:30.730+0000 7fd15b0af4c0 -1 osd.1 31 log_to_monitors true 2024-02-05T05:46:30.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:31.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:31.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:31.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:46:31.945 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:46:31.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:31.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:46:32.345 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1239454348,v1:127.0.0.1:6811/1239454348] [v2:127.0.0.1:6812/1239454348,v1:127.0.0.1:6813/1239454348] exists,up 1ff70854-a136-4e5c-b63b-f7e5b0c9a2f2 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:46:32.346 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:46:32.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:46:32.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:46:32.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:46:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:46:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:46:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:46:32.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:46:32.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:46:32.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:46:32.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:46:32.799 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:46:32.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:46:32.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:32.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:46:32.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-02-05T05:46:32.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-02-05T05:46:32.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-02-05T05:46:32.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:32.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:46:33.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-02-05T05:46:33.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-02-05T05:46:33.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-150323855362' 2024-02-05T05:46:33.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:33.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-02-05T05:46:33.005 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:33.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:46:33.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-02-05T05:46:33.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:33.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-02-05T05:46:33.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-02-05T05:46:33.008 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 21474836497 2024-02-05T05:46:33.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:46:33.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836497 2024-02-05T05:46:33.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:46:34.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:46:34.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:46:34.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-02-05T05:46:34.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:34.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-02-05T05:46:34.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:34.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:46:34.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-02-05T05:46:34.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:34.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-02-05T05:46:34.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-02-05T05:46:34.553 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 150323855362 2024-02-05T05:46:34.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:46:34.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-02-05T05:46:34.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:46:34.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:34.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:35.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:46:35.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:46:35.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:46:35.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:46:35.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:46:35.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:46:35.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:46:35.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:46:35.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:46:35.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:46:35.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:35.417 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:46:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:46:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:46:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-02-05T05:46:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-02-05T05:46:35.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-02-05T05:46:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-02-05T05:46:35.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-02-05T05:46:35.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:46:35.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:46:35.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:46:35.876 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:46:35.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:46:35.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:46:35.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:46:36.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:46:36.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-02-05T05:46:36.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:46:36.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:46:36.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:46:36.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:46:36.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:46:36.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-02-05T05:46:38.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T05:46:38.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:46:38.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:46:38.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:46:38.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:46:38.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:46:38.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:46:38.035 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:46:38.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:46:38.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:46:38.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:46:38.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:46:38.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:46:38.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:46:38.062 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:38.069+0000 7f7fbb4164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:38.064 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:38.072+0000 7f7fbb4164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:38.065 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:38.073+0000 7f7fbb4164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:38.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:46:38.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:46:38.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:46:38.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:46:38.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:46:38.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:38.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:46:38.349 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:46:38.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:38.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:46:38.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:38.585 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:38.593+0000 7f7fbb4164c0 -1 Falling back to public interface 2024-02-05T05:46:39.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:39.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:39.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:46:39.578 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:46:39.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:39.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:46:39.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:46:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:46:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:40.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:46:40.871 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:40.879+0000 7f7fbb4164c0 -1 osd.0 37 log_to_monitors true 2024-02-05T05:46:41.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:42.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:42.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:42.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:46:42.210 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:46:42.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:42.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:46:42.485 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/163851588,v1:127.0.0.1:6803/163851588] [v2:127.0.0.1:6804/163851588,v1:127.0.0.1:6805/163851588] exists,up 038b4cb7-21f1-477e-8dca-d018fa778d2f 2024-02-05T05:46:42.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:46:42.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:46:42.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:46:42.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:46:42.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:46:42.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:46:42.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:46:42.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:46:42.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:46:42.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:46:42.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:46:42.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:46:42.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:46:42.840 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:46:42.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:46:42.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:42.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:46:42.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-02-05T05:46:42.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-02-05T05:46:42.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659138' 2024-02-05T05:46:42.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:46:42.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:46:43.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-02-05T05:46:43.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-02-05T05:46:43.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659138 1-150323855365' 2024-02-05T05:46:43.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:43.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659138 2024-02-05T05:46:43.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:46:43.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659138 2024-02-05T05:46:43.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-02-05T05:46:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659138' 2024-02-05T05:46:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 176093659138 2024-02-05T05:46:43.051 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:46:43.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-02-05T05:46:43.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:46:43.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-02-05T05:46:43.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:46:43.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:46:43.342 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-02-05T05:46:43.342 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:46:43.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-02-05T05:46:43.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-02-05T05:46:43.343 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 150323855365 2024-02-05T05:46:43.343 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:46:43.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-02-05T05:46:43.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:46:43.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:43.614 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:43.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T05:46:43.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:46:43.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:46:43.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:46:43.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:46:43.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:46:43.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:46:43.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:46:44.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T05:46:44.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:46:44.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:46:44.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:46:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T05:46:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:46:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:46:44.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-02-05T05:46:44.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-02-05T05:46:44.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-02-05T05:46:44.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-02-05T05:46:44.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:46:44.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:46:44.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:46:44.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:46:44.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T05:46:14.459743+0000 2024-02-05T05:46:44.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-02-05T05:46:45.070 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 3.0 on osd.1 to scrub 2024-02-05T05:46:45.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-02-05T05:46:14.459743+0000 2024-02-05T05:46:45.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:46:14.459743+0000 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:46:45.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:46:45.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:46:14.459743+0000 '>' 2024-02-05T05:46:14.459743+0000 2024-02-05T05:46:45.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:46:46.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:46:46.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:46:46.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:46:46.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:46:46.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:46:46.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:46:46.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:46:46.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:46:14.459743+0000 '>' 2024-02-05T05:46:14.459743+0000 2024-02-05T05:46:46.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:46:47.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:46:47.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:46:47.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-02-05T05:46:47.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-02-05T05:46:47.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:46:47.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:46:47.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-02-05T05:46:47.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:46:45.848918+0000 '>' 2024-02-05T05:46:14.459743+0000 2024-02-05T05:46:47.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:46:47.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-02-05T05:46:47.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-02-05T05:46:47.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-02-05T05:46:47.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-02-05T05:46:47.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-02-05T05:46:47.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-02-05T05:46:47.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-02-05T05:46:47.948 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:46:47.948 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:46:47.948 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-02-05T05:46:47.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr: . as $in 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr: | if type == "object" then 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr: reduce keys[] as $key 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr: elif type == "array" then map( walk(f) ) | f 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr: else f 2024-02-05T05:46:47.971 INFO:tasks.workunit.client.0.smithi188.stderr: end; 2024-02-05T05:46:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-02-05T05:46:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-02-05T05:46:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-02-05T05:46:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.version) else . end) 2024-02-05T05:46:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-02-05T05:46:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-02-05T05:46:47.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-02-05T05:46:47.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:46:47.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-02-05T05:46:47.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-02-05T05:46:47.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T05:46:47.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-02-05T05:46:47.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T05:46:47.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-02-05T05:46:47.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T05:46:47.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-02-05T05:46:48.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T05:46:48.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-02-05T05:46:48.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T05:46:48.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-02-05T05:46:48.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-02-05T05:46:48.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-02-05T05:46:48.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-02-05T05:46:48.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-02-05T05:46:48.368 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'csr_pool' removed 2024-02-05T05:46:48.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:46:48.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:46:48.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:46:48.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:46:48.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:46:48.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:46:48.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:46:48.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:46:48.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:46:48.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:46:48.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:46:48.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:46:48.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:46:48.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:46:48.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:46:48.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:46:48.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:46:48.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:46:48.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:46:48.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:46:48.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:46:48.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:46:48.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:46:48.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:46:48.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:48.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:48.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:46:48.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:46:48.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:46:48.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:46:48.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:46:48.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:46:48.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:46:48.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:46:48.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:46:48.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:46:48.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:46:48.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:46:48.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:46:48.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:46:48.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:46:48.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:46:48.514 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:46:48.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:46:48.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:46:48.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:46:48.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:46:48.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:48.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:48.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:46:48.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:46:48.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:46:48.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:46:48.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:46:48.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:48.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:48.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:46:48.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:46:48.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:46:48.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:46:48.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:46:48.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-02-05T05:46:48.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-02-05T05:46:48.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:46:48.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:48.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:46:49.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:46:49.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:46:49.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:46:49.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:46:49.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:46:49.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:46:49.080 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:46:49.080 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:46:49.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:46:49.081 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:46:49.081 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:49.081 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:49.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:46:49.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:46:49.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:46:49.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:46:49.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:46:49.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:46:49.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:46:49.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:46:49.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:46:49.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T05:46:49.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:46:49.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:46:49.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:46:49.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:46:49.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:46:49.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:46:50.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:46:50.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:46:50.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:46:50.045 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:46:50.045 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:50.045 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:50.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:50.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:46:50.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:46:50.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-02-05T05:46:50.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:46:50.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T05:46:50.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:46:50.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:46:50.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:46:50.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:46:50.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:46:50.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:46:50.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:46:50.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:46:50.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93940848-8f7b-4689-b430-dc2cfdedabf6 2024-02-05T05:46:50.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 93940848-8f7b-4689-b430-dc2cfdedabf6' 2024-02-05T05:46:50.081 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 93940848-8f7b-4689-b430-dc2cfdedabf6 2024-02-05T05:46:50.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:46:50.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKdsBlZ9g9BhAAtbEhgK7DCNGigAhcMYe46g== 2024-02-05T05:46:50.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKdsBlZ9g9BhAAtbEhgK7DCNGigAhcMYe46g=="}' 2024-02-05T05:46:50.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93940848-8f7b-4689-b430-dc2cfdedabf6 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:46:50.301 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:46:50.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:46:50.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKdsBlZ9g9BhAAtbEhgK7DCNGigAhcMYe46g== --osd-uuid 93940848-8f7b-4689-b430-dc2cfdedabf6 2024-02-05T05:46:50.339 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:50.348+0000 7feca80164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:50.352 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:50.361+0000 7feca80164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:50.353 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:50.362+0000 7feca80164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:50.353 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:50.362+0000 7feca80164c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:46:50.354 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:50.362+0000 7feca80164c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:46:50.354 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:50.362+0000 7feca80164c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:46:50.428 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:50.436+0000 7feca80164c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:46:55.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:46:55.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:46:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:46:55.768 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:46:55.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:46:56.146 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:46:56.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:46:56.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:46:56.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:46:56.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:46:56.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:46:56.173 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:56.181+0000 7fd3bea154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:56.175 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:56.184+0000 7fd3bea154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:56.176 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:56.185+0000 7fd3bea154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:46:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:46:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:46:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:46:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:46:56.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:46:56.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:56.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:46:56.417 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:46:56.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:56.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:46:56.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:56.698 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:56.707+0000 7fd3bea154c0 -1 Falling back to public interface 2024-02-05T05:46:57.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:57.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:57.684 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:46:57.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:46:57.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:57.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:46:57.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:46:58.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:46:58.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:46:58.961 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:46:58.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:46:58.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:46:58.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:46:59.034 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:46:59.043+0000 7fd3bea154c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:46:59.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:00.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:00.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:00.271 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:47:00.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:47:00.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:00.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:00.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:01.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:01.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:01.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:47:01.609 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:47:01.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:01.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:02.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:03.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:47:03.006 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:47:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:03.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:03.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:04.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:04.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:04.286 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:47:04.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:47:04.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:04.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:04.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:05.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:05.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:05.674 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:47:05.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:47:05.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:05.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:05.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:06.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:06.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:06.963 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:47:06.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:47:06.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:06.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:07.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:08.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:08.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:08.241 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:47:08.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:47:08.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:08.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:08.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:09.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:09.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:09.522 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:47:09.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:47:09.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:09.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:09.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:10.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:10.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:10.800 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:47:10.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:47:10.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:10.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:11.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:12.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:12.283 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:47:12.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:12.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:47:12.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:12.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:12.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:13.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:13.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:13.943 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:47:13.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:47:13.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:13.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:14.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:15.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:15.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:15.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:47:15.576 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:47:15.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:15.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:16.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:17.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:17.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:17.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:47:17.132 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:47:17.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:17.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:17.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:18.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:18.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:18.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:47:18.827 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:47:18.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:18.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:19.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:20.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:20.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:20.546 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:47:20.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:47:20.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:20.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:47:20.883 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3428898303,v1:127.0.0.1:6803/3428898303] [v2:127.0.0.1:6804/3428898303,v1:127.0.0.1:6805/3428898303] exists,up 93940848-8f7b-4689-b430-dc2cfdedabf6 2024-02-05T05:47:20.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:47:20.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:47:20.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:47:20.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:47:20.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:47:20.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:47:20.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d361615c-8419-421b-b326-1b1ad340dd46 2024-02-05T05:47:20.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d361615c-8419-421b-b326-1b1ad340dd46' 2024-02-05T05:47:20.891 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 d361615c-8419-421b-b326-1b1ad340dd46 2024-02-05T05:47:20.891 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:47:20.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBodsBlgamLNhAAzhjfEpmlBkyYBdTZPUwaSg== 2024-02-05T05:47:20.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBodsBlgamLNhAAzhjfEpmlBkyYBdTZPUwaSg=="}' 2024-02-05T05:47:20.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d361615c-8419-421b-b326-1b1ad340dd46 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:47:21.265 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:47:21.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:47:21.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBodsBlgamLNhAAzhjfEpmlBkyYBdTZPUwaSg== --osd-uuid d361615c-8419-421b-b326-1b1ad340dd46 2024-02-05T05:47:21.305 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:21.314+0000 7fe1d26cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:21.341 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:21.350+0000 7fe1d26cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:21.342 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:21.351+0000 7fe1d26cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:21.342 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:21.352+0000 7fe1d26cf4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:47:21.342 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:21.352+0000 7fe1d26cf4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:47:21.342 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:21.352+0000 7fe1d26cf4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:47:21.408 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:21.417+0000 7fe1d26cf4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:47:27.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:47:27.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:47:27.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:47:27.190 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:47:27.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:47:27.535 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:47:27.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:47:27.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:47:27.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:47:27.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:47:27.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:47:27.562 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:27.571+0000 7f0e0521a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:27.563 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:27.573+0000 7f0e0521a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:27.565 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:27.574+0000 7f0e0521a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:27.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:47:27.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:47:27.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:47:27.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:47:27.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:47:27.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:27.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:47:27.805 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:47:27.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:27.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:28.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:28.093 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:28.103+0000 7f0e0521a4c0 -1 Falling back to public interface 2024-02-05T05:47:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:29.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:29.074 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:47:29.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:47:29.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:29.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:29.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:30.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:30.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:30.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:47:30.351 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:47:30.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:30.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:30.524 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:30.533+0000 7f0e0521a4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:47:30.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:31.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:31.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:31.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:47:31.631 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:47:31.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:31.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:31.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:32.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:32.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:32.912 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:47:32.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:47:32.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:32.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:33.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:34.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:34.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:34.260 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:47:34.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:47:34.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:34.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:34.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:35.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:35.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:35.559 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:47:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:47:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:35.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:36.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:36.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:36.836 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:47:36.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:47:36.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:36.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:37.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:38.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:38.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:38.114 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:47:38.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:47:38.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:38.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:38.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:39.502 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:47:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:47:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:39.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:40.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:40.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:40.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:47:40.783 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:47:40.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:40.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:41.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:42.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:42.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:42.063 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:47:42.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:47:42.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:42.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:42.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:43.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:43.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:43.364 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:47:43.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:47:43.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:43.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:43.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:44.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:44.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:44.892 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:47:44.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:47:44.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:44.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:45.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:46.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:46.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:46.424 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:47:46.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:47:46.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:46.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:46.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:47.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:47.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:47.783 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:47:47.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:47:47.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:47.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:48.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:49.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:49.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:49.202 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:47:49.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:47:49.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:49.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:49.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:50.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:50.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:50.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:47:50.964 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:47:50.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:50.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:51.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:52.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:52.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:52.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:47:52.537 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:47:52.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:52.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:52.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:47:53.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:47:53.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:47:53.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:47:53.818 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:47:53.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:47:53.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:47:54.096 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1218253669,v1:127.0.0.1:6811/1218253669] [v2:127.0.0.1:6812/1218253669,v1:127.0.0.1:6813/1218253669] exists,up d361615c-8419-421b-b326-1b1ad340dd46 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:47:54.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:47:54.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26c9e623-811c-4364-b593-e9ef856746f8 2024-02-05T05:47:54.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 26c9e623-811c-4364-b593-e9ef856746f8' 2024-02-05T05:47:54.104 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 26c9e623-811c-4364-b593-e9ef856746f8 2024-02-05T05:47:54.104 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:47:54.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKdsBllUfBBxAA1C6EXLNjPVZsygt0hZgA8g== 2024-02-05T05:47:54.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKdsBllUfBBxAA1C6EXLNjPVZsygt0hZgA8g=="}' 2024-02-05T05:47:54.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26c9e623-811c-4364-b593-e9ef856746f8 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:47:54.453 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:47:54.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:47:54.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKdsBllUfBBxAA1C6EXLNjPVZsygt0hZgA8g== --osd-uuid 26c9e623-811c-4364-b593-e9ef856746f8 2024-02-05T05:47:54.494 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:54.504+0000 7f263f40c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:54.538 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:54.548+0000 7f263f40c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:54.540 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:54.549+0000 7f263f40c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:47:54.540 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:54.549+0000 7f263f40c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:47:54.540 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:54.549+0000 7f263f40c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:47:54.540 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:54.549+0000 7f263f40c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:47:54.613 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:47:54.623+0000 7f263f40c4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:48:00.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:48:00.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:48:00.330 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:48:00.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:48:00.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:48:00.677 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:48:00.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:48:00.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:48:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:48:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:48:00.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:48:00.703 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:00.713+0000 7f7787e1f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:00.705 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:00.715+0000 7f7787e1f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:00.706 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:00.717+0000 7f7787e1f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:00.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:48:00.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:48:00.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:48:00.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:48:00.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:48:00.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:00.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:48:00.948 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:48:00.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:00.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:01.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:01.233 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:01.244+0000 7f7787e1f4c0 -1 Falling back to public interface 2024-02-05T05:48:02.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:02.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:02.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:48:02.215 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:48:02.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:02.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:02.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:03.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:03.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:03.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:48:03.485 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:48:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:03.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:03.588 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:03.598+0000 7f7787e1f4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:48:03.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:04.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:04.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:04.798 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:48:04.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:48:04.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:04.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:05.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:06.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:06.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:06.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:48:06.073 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:48:06.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:06.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:06.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:07.443 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:48:07.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:48:07.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:07.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:07.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:08.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:48:08.721 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:48:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:08.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:08.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:09.999 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:48:09.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:48:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:10.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:10.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:11.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:11.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:11.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:48:11.468 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:48:11.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:11.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:11.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:12.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:12.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:12.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:48:12.749 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:48:12.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:12.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:13.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:14.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:14.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:14.028 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:48:14.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:48:14.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:14.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:14.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:15.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:15.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:15.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:48:15.402 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:48:15.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:15.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:15.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:16.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:16.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:16.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:48:16.680 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:48:16.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:16.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:17.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:18.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:18.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:18.019 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:48:18.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:48:18.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:18.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:18.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:19.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:19.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:19.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:48:19.427 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:48:19.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:19.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:19.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:20.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:20.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:20.997 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:48:20.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:48:20.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:20.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:21.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:22.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:22.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:22.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:48:22.767 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:48:22.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:22.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:23.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:24.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:24.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:24.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:48:24.109 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:48:24.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:24.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:24.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:25.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:25.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:25.897 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:48:25.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:48:25.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:25.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:26.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:27.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:27.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:27.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:48:27.265 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:48:27.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:27.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:48:27.538 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1450884398,v1:127.0.0.1:6819/1450884398] [v2:127.0.0.1:6820/1450884398,v1:127.0.0.1:6821/1450884398] exists,up 26c9e623-811c-4364-b593-e9ef856746f8 2024-02-05T05:48:27.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:48:27.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:48:27.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:48:27.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-02-05T05:48:27.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:48:27.793 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:48:27.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:48:27.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:48:28.200 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:48:28.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:48:29.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:48:30.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-02-05T05:48:30.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:48:30.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:48:30.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:48:30.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:48:30.959 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:48:30.959 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:48:30.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:48:30.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:48:30.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:48:31.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:48:31.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:48:31.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:48:31.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:48:31.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:48:31.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:48:31.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:48:31.323 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:48:31.323 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:48:31.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:48:31.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:48:31.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:48:31.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-02-05T05:48:31.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-02-05T05:48:31.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-02-05T05:48:31.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:48:31.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:48:31.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T05:48:31.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T05:48:31.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972' 2024-02-05T05:48:31.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:48:31.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:48:31.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-02-05T05:48:31.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-02-05T05:48:31.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672972 2-64424509448' 2024-02-05T05:48:31.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:48:31.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-02-05T05:48:31.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:48:31.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:48:31.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-02-05T05:48:31.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:48:31.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-02-05T05:48:31.649 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836501 2024-02-05T05:48:31.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-02-05T05:48:31.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:48:31.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-02-05T05:48:31.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:48:32.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:48:32.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:48:33.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836501 2024-02-05T05:48:33.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:48:33.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T05:48:33.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:48:33.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:48:33.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T05:48:33.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:48:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T05:48:33.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T05:48:33.193 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672972 2024-02-05T05:48:33.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:48:33.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T05:48:33.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:48:33.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-02-05T05:48:33.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:48:33.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:48:33.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-02-05T05:48:33.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:48:33.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-02-05T05:48:33.504 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509448 2024-02-05T05:48:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-02-05T05:48:33.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:48:33.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-02-05T05:48:33.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:48:33.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:48:33.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:48:34.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:48:34.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:48:34.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:48:34.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:48:34.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:48:34.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:48:34.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:48:34.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:48:34.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:48:34.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:48:34.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:48:34.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:48:34.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T05:48:35.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:48:35.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:48:35.656 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:48:35.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:48:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:48:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:48:36.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:48:36.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:48:36.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:48:36.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:48:36.671 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:48:36.671 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:48:36.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:48:36.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:48:36.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:48:36.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:48:36.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:48:36.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:48:36.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:48:36.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:48:36.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:48:37.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:48:37.050 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:48:37.050 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:48:37.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:48:37.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:48:37.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:48:37.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-05T05:48:37.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-05T05:48:37.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-05T05:48:37.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:48:37.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:48:37.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-05T05:48:37.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-05T05:48:37.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672974' 2024-02-05T05:48:37.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:48:37.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:48:37.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-02-05T05:48:37.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-02-05T05:48:37.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672974 2-64424509451' 2024-02-05T05:48:37.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:48:37.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-05T05:48:37.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:48:37.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:48:37.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-05T05:48:37.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:48:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-05T05:48:37.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-05T05:48:37.422 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836504 2024-02-05T05:48:37.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:48:37.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-02-05T05:48:37.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:48:38.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:48:38.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:48:38.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-02-05T05:48:38.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:48:38.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-05T05:48:38.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:48:38.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:48:38.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-05T05:48:38.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:48:38.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-05T05:48:38.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-05T05:48:38.973 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672974 2024-02-05T05:48:38.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:48:39.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-02-05T05:48:39.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:48:39.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-02-05T05:48:39.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:48:39.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:48:39.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-02-05T05:48:39.241 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:48:39.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-02-05T05:48:39.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-02-05T05:48:39.242 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509451 2024-02-05T05:48:39.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:48:39.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-02-05T05:48:39.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:48:39.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:48:39.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:48:39.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:48:39.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:48:39.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:48:39.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:48:39.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:48:39.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:48:39.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:48:39.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:48:40.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:48:40.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:48:40.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:48:40.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:48:40.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:48:40.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:48:40.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:48:40.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:48:40.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-02-05T05:48:40.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:48:40.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:48:40.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-02-05T05:48:40.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:48:40.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:48:40.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:48:41.142 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:48:41.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:48:41.519 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:48:41.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:48:41.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:48:41.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:48:41.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-02-05T05:48:41.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:48:41.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-02-05T05:48:41.817 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-02-05T05:48:41.817 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:48:42.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T05:48:42.089 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:48:42.089 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:48:42.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T05:48:42.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-02-05T05:48:42.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-02-05T05:48:42.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-02-05T05:48:42.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:48:42.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:48:42.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-02-05T05:48:42.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:48:42.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:48:42.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:48:42.474 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:48:42.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:48:42.850 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:48:42.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:48:42.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:48:42.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:48:43.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-02-05T05:48:43.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:48:43.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-02-05T05:48:43.123 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-02-05T05:48:43.123 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:48:43.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T05:48:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:48:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:48:43.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T05:48:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-02-05T05:48:43.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-02-05T05:48:43.399 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-02-05T05:48:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:48:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:48:43.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:48:43.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:48:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:48:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:48:43.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:48:43.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:48:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:48:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:48:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:48:44.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:48:45.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:48:45.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-02-05T05:48:45.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-02-05T05:48:46.509 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-02-05T05:48:47.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-02-05T05:48:47.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-02-05T05:48:48.114 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-02-05T05:48:48.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-02-05T05:48:48.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-02-05T05:48:49.316 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-02-05T05:48:49.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-02-05T05:48:49.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-02-05T05:48:51.046 INFO:tasks.workunit.client.0.smithi188.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-02-05T05:48:51.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:48:51.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:48:51.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:48:51.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:48:51.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:48:51.573 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:48:51.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:48:51.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:48:51.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:48:51.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:48:51.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:48:51.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:48:51.600 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:51.611+0000 7f499f5e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:51.602 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:51.613+0000 7f499f5e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:51.603 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:51.615+0000 7f499f5e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:48:52.117 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:52.129+0000 7f499f5e24c0 -1 Falling back to public interface 2024-02-05T05:48:52.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:53.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:53.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:53.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:48:53.229 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:48:53.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:53.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:48:53.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:54.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:54.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:54.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:48:54.502 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:48:54.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:54.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:48:54.522 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:54.534+0000 7f499f5e24c0 -1 osd.0 33 log_to_monitors true 2024-02-05T05:48:54.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:55.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:55.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:55.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:48:55.809 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:48:55.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:55.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:48:56.089 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/3635118899,v1:127.0.0.1:6803/3635118899] [v2:127.0.0.1:6804/3635118899,v1:127.0.0.1:6805/3635118899] exists,up 93940848-8f7b-4689-b430-dc2cfdedabf6 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:48:56.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:48:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:48:56.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:48:56.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:48:56.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:48:56.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:48:56.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:48:56.114 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:56.126+0000 7faf73a054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:56.150 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:56.162+0000 7faf73a054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:56.151 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:56.163+0000 7faf73a054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:48:56.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:48:56.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:48:56.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:48:56.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:48:56.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:48:56.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:56.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:48:56.360 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:48:56.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:56.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:48:56.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:56.665 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:56.677+0000 7faf73a054c0 -1 Falling back to public interface 2024-02-05T05:48:57.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:57.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:57.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:48:57.626 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:48:57.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:57.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:48:57.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:48:58.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:48:58.901+0000 7faf73a054c0 -1 osd.1 34 log_to_monitors true 2024-02-05T05:48:58.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:48:58.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:48:58.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:48:58.898 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:48:58.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:48:58.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:48:59.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:00.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:00.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:00.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:49:00.178 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:49:00.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:00.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:49:00.478 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 35 last_clean_interval [10,34) [v2:127.0.0.1:6810/3002806739,v1:127.0.0.1:6811/3002806739] [v2:127.0.0.1:6812/3002806739,v1:127.0.0.1:6813/3002806739] exists,up d361615c-8419-421b-b326-1b1ad340dd46 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:49:00.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:00.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.2 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:49:00.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:49:00.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:49:00.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:49:00.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:49:00.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:49:00.504 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:00.515+0000 7f1064c0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:00.505 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:00.517+0000 7f1064c0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:00.507 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:00.518+0000 7f1064c0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:00.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:49:01.025 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:01.037+0000 7f1064c0d4c0 -1 Falling back to public interface 2024-02-05T05:49:01.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:02.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:02.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:02.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:49:02.053 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:49:02.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:02.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:49:02.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:03.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:03.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:03.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:49:03.333 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:49:03.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:03.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:49:03.374 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:03.385+0000 7f1064c0d4c0 -1 osd.2 36 log_to_monitors true 2024-02-05T05:49:03.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:04.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:04.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:04.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:49:04.713 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:49:04.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:04.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:49:05.042 INFO:tasks.workunit.client.0.smithi188.stderr:osd.2 up in weight 1 up_from 46 up_thru 46 down_at 37 last_clean_interval [15,36) [v2:127.0.0.1:6818/892201174,v1:127.0.0.1:6819/892201174] [v2:127.0.0.1:6820/892201174,v1:127.0.0.1:6821/892201174] exists,up 26c9e623-811c-4364-b593-e9ef856746f8 2024-02-05T05:49:05.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:49:05.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:49:05.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:49:05.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-02-05T05:49:05.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:49:05.443 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' removed 2024-02-05T05:49:05.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:49:05.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:49:06.037 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:49:06.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:49:07.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:49:08.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-02-05T05:49:08.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:49:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:49:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:49:08.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:49:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:49:08.810 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:49:08.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:49:08.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:49:08.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:49:08.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:49:08.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:49:08.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:49:08.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:49:08.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:49:08.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:49:09.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:49:09.177 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:49:09.177 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:49:09.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:49:09.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:49:09.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:49:09.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724548 2024-02-05T05:49:09.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724548 2024-02-05T05:49:09.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724548' 2024-02-05T05:49:09.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:49:09.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:49:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626436 2024-02-05T05:49:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626436 2024-02-05T05:49:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724548 1-180388626436' 2024-02-05T05:49:09.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:49:09.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:49:09.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495619 2024-02-05T05:49:09.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495619 2024-02-05T05:49:09.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-167503724548 1-180388626436 2-197568495619' 2024-02-05T05:49:09.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:49:09.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-167503724548 2024-02-05T05:49:09.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:49:09.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:49:09.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-167503724548 2024-02-05T05:49:09.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:49:09.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724548 2024-02-05T05:49:09.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 167503724548' 2024-02-05T05:49:09.488 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 167503724548 2024-02-05T05:49:09.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:49:09.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724547 -lt 167503724548 2024-02-05T05:49:09.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:49:10.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:49:10.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:49:11.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724548 -lt 167503724548 2024-02-05T05:49:11.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:49:11.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-180388626436 2024-02-05T05:49:11.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:49:11.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:49:11.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-180388626436 2024-02-05T05:49:11.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:49:11.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626436 2024-02-05T05:49:11.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 180388626436' 2024-02-05T05:49:11.043 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 180388626436 2024-02-05T05:49:11.043 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:49:11.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626436 -lt 180388626436 2024-02-05T05:49:11.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:49:11.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-197568495619 2024-02-05T05:49:11.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:49:11.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:49:11.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-197568495619 2024-02-05T05:49:11.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:49:11.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495619 2024-02-05T05:49:11.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-02-05T05:49:11.315 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 197568495619 2024-02-05T05:49:11.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:49:11.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495619 -lt 197568495619 2024-02-05T05:49:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:49:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:49:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:49:11.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:49:11.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:49:11.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:49:11.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:49:11.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:49:11.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:49:11.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:49:11.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:49:12.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:49:12.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:49:12.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:49:12.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:49:12.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:49:12.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:49:12.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:49:12.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-02-05T05:49:12.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:49:12.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-02-05T05:49:12.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-02-05T05:49:12.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:49:12.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-02-05T05:49:12.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-02-05T05:49:12.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:49:12.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:49:12.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:49:12.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:49:12.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:49:12.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:49:13.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:48:35.592169+0000 2024-02-05T05:49:13.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:49:13.299 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-02-05T05:49:13.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:48:35.592169+0000 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:48:35.592169+0000 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:49:13.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:49:13.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:48:35.592169+0000 '>' 2024-02-05T05:48:35.592169+0000 2024-02-05T05:49:13.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:49:14.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:49:14.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:49:14.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:49:14.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:49:14.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:49:14.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:49:14.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:49:14.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:49:13.547191+0000 '>' 2024-02-05T05:48:35.592169+0000 2024-02-05T05:49:14.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:49:14.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-02-05T05:49:14.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-02-05T05:49:14.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-02-05T05:49:14.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-02-05T05:49:14.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-02-05T05:49:14.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:49:14.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:49:14.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:49:14.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:49:14.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:49:14.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:49:15.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:49:15.066 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:49:15.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:49:15.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:49:15.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:49:15.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:49:15.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:49:15.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:49:15.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:49:15.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:49:15.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:49:15.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:49:15.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:49:15.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:49:15.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:49:15.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:15.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:15.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:49:15.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:49:15.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:49:15.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:49:15.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:49:15.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:49:15.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:49:15.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:49:15.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:49:15.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:49:15.094 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:49:15.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:49:15.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:49:15.095 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:49:15.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:49:15.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:49:15.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:49:15.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:49:15.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:49:15.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:49:15.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:49:15.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:49:15.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:15.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:15.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:49:15.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:49:15.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:49:15.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:49:15.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:49:15.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:15.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:15.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:49:15.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:49:15.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:49:15.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:49:15.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:49:15.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:49:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:49:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:49:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:49:15.481 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:49:15.481 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:15.481 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:15.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:15.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:49:15.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:49:15.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:49:15.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:49:15.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:49:15.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:49:15.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:49:15.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:49:15.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:49:15.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:49:15.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:49:15.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:49:15.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:49:15.749 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:49:15.749 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:49:15.749 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:49:15.750 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:49:15.750 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:15.750 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:15.750 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:49:15.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:49:15.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:49:15.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T05:49:15.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:49:15.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:49:15.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:49:15.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:49:15.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:49:15.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:49:16.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:49:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:49:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:49:16.635 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:49:16.635 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:16.635 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:16.635 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:49:16.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:49:16.663 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-02-05T05:49:16.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:49:16.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T05:49:16.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:49:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:49:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:49:16.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=507968b6-910b-45e1-945e-948771d82a01 2024-02-05T05:49:16.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 507968b6-910b-45e1-945e-948771d82a01' 2024-02-05T05:49:16.671 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 507968b6-910b-45e1-945e-948771d82a01 2024-02-05T05:49:16.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:49:16.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDcdsBl9GWhKRAAHDdkrcaPVQt4Q9h+plXWhA== 2024-02-05T05:49:16.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDcdsBl9GWhKRAAHDdkrcaPVQt4Q9h+plXWhA=="}' 2024-02-05T05:49:16.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 507968b6-910b-45e1-945e-948771d82a01 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:49:16.907 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:49:16.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:49:16.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDcdsBl9GWhKRAAHDdkrcaPVQt4Q9h+plXWhA== --osd-uuid 507968b6-910b-45e1-945e-948771d82a01 2024-02-05T05:49:16.946 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:16.958+0000 7f597d4a54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:16.958 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:16.970+0000 7f597d4a54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:16.959 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:16.971+0000 7f597d4a54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:16.959 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:16.972+0000 7f597d4a54c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:49:16.959 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:16.972+0000 7f597d4a54c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:49:16.959 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:16.972+0000 7f597d4a54c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:49:17.050 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:17.062+0000 7f597d4a54c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:49:22.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:49:22.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:49:22.590 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:49:22.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:49:22.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:49:22.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:49:22.935 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:49:22.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:49:22.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:49:22.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:49:22.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:49:22.961 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:22.973+0000 7f1c6159f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:22.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:22.975+0000 7f1c6159f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:22.964 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:22.976+0000 7f1c6159f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:49:23.201 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:49:23.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:23.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:23.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:23.486 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:23.498+0000 7f1c6159f4c0 -1 Falling back to public interface 2024-02-05T05:49:24.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:24.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:24.470 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:49:24.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:49:24.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:24.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:24.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:25.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:25.747 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:49:25.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:25.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:49:25.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:25.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:25.866 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:25.878+0000 7f1c6159f4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:49:26.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:27.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:27.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:27.025 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:49:27.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:49:27.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:27.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:27.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:28.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:28.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:28.307 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:49:28.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:49:28.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:28.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:28.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:29.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:29.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:29.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:49:29.674 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:49:29.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:29.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:29.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:30.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:30.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:30.949 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:49:30.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:49:30.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:30.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:31.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:32.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:32.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:32.224 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:49:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:49:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:32.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:33.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:33.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:33.505 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:49:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:49:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:33.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:33.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:34.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:34.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:34.834 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:49:34.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:49:34.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:34.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:35.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:36.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:36.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:36.111 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:49:36.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:49:36.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:36.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:36.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:37.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:37.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:37.391 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:49:37.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:49:37.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:37.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:37.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:38.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:38.811 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:49:38.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:38.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:49:38.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:38.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:39.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:40.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:40.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:40.143 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:49:40.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:49:40.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:40.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:40.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:41.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:41.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:41.694 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:49:41.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:49:41.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:41.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:42.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:43.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:43.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:43.439 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:49:43.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:49:43.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:43.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:44.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:45.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:45.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:45.056 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:49:45.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:49:45.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:45.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:45.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:46.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:46.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:46.836 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:49:46.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:49:46.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:46.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:47.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:48.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:48.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:48.333 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:49:48.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:49:48.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:48.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:49:48.658 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3720610092,v1:127.0.0.1:6803/3720610092] [v2:127.0.0.1:6804/3720610092,v1:127.0.0.1:6805/3720610092] exists,up 507968b6-910b-45e1-945e-948771d82a01 2024-02-05T05:49:48.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:49:48.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:49:48.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:49:48.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:49:48.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea5cb4fc-03f2-4640-833f-2940ac9bd8bd 2024-02-05T05:49:48.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ea5cb4fc-03f2-4640-833f-2940ac9bd8bd' 2024-02-05T05:49:48.666 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 ea5cb4fc-03f2-4640-833f-2940ac9bd8bd 2024-02-05T05:49:48.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:49:48.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8dsBl1cJiKRAApuKVXNBLsnGQ4EXXjthT0A== 2024-02-05T05:49:48.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8dsBl1cJiKRAApuKVXNBLsnGQ4EXXjthT0A=="}' 2024-02-05T05:49:48.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea5cb4fc-03f2-4640-833f-2940ac9bd8bd -i td/osd-scrub-repair/1/new.json 2024-02-05T05:49:49.099 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:49:49.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:49:49.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8dsBl1cJiKRAApuKVXNBLsnGQ4EXXjthT0A== --osd-uuid ea5cb4fc-03f2-4640-833f-2940ac9bd8bd 2024-02-05T05:49:49.140 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:49.152+0000 7fb33d4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:49.184 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:49.197+0000 7fb33d4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:49.198+0000 7fb33d4144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:49.198+0000 7fb33d4144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:49:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:49.198+0000 7fb33d4144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:49:49.185 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:49.198+0000 7fb33d4144c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:49:49.259 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:49.272+0000 7fb33d4144c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:49:55.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:49:55.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:49:55.193 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:49:55.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:49:55.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:49:55.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:49:55.540 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:49:55.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:49:55.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:49:55.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:49:55.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:49:55.566 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:55.579+0000 7f7e420144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:55.568 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:55.581+0000 7f7e420144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:55.569 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:55.582+0000 7f7e420144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:49:55.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:49:55.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:49:55.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:49:55.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:49:55.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:49:55.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:55.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:49:55.812 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:49:55.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:55.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:49:56.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:56.091 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:56.104+0000 7f7e420144c0 -1 Falling back to public interface 2024-02-05T05:49:57.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:57.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:57.079 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:49:57.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:49:57.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:57.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:49:57.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:58.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:58.348 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:49:58.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:58.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:49:58.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:58.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:49:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:49:58.421+0000 7f7e420144c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:49:58.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:49:59.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:49:59.620 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:49:59.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:49:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:49:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:49:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:49:59.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:00.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:00.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:00.900 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:50:00.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:50:00.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:00.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:01.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:02.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:02.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:02.227 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:50:02.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:50:02.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:02.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:02.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:03.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:03.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:03.500 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:50:03.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:50:03.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:03.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:03.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:04.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:04.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:04.778 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:50:04.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:50:04.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:04.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:05.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:06.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:06.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:06.185 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:50:06.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:50:06.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:06.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:06.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:07.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:07.492 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:50:07.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:07.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:50:07.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:07.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:07.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:08.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:08.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:08.772 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:50:08.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:50:08.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:08.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:09.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:10.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:10.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:10.063 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:50:10.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:50:10.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:10.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:10.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:11.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:11.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:11.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:50:11.541 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:50:11.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:11.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:11.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:12.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:12.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:12.919 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:50:12.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:50:12.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:12.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:13.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:14.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:14.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:14.436 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:50:14.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:50:14.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:14.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:15.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:16.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:16.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:16.049 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:50:16.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:50:16.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:16.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:16.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:17.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:17.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:17.887 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:50:17.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:50:17.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:17.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:18.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:19.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:19.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:19.735 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:50:19.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:50:19.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:19.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:20.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:21.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:21.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:21.283 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:50:21.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:50:21.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:21.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:21.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:22.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:22.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:22.564 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:50:22.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:50:22.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:22.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:50:22.856 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3175533759,v1:127.0.0.1:6811/3175533759] [v2:127.0.0.1:6812/3175533759,v1:127.0.0.1:6813/3175533759] exists,up ea5cb4fc-03f2-4640-833f-2940ac9bd8bd 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:50:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:50:22.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:50:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=880dce46-8ffb-4945-a247-948346d2bc8e 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 880dce46-8ffb-4945-a247-948346d2bc8e' 2024-02-05T05:50:22.863 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 880dce46-8ffb-4945-a247-948346d2bc8e 2024-02-05T05:50:22.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:50:22.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAed8BlM70kNRAAwRb0FhnJV1rWZFfRYFWKOg== 2024-02-05T05:50:22.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAed8BlM70kNRAAwRb0FhnJV1rWZFfRYFWKOg=="}' 2024-02-05T05:50:22.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 880dce46-8ffb-4945-a247-948346d2bc8e -i td/osd-scrub-repair/2/new.json 2024-02-05T05:50:23.245 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:50:23.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:50:23.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAed8BlM70kNRAAwRb0FhnJV1rWZFfRYFWKOg== --osd-uuid 880dce46-8ffb-4945-a247-948346d2bc8e 2024-02-05T05:50:23.285 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:23.299+0000 7fce3b61f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:23.337 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:23.351+0000 7fce3b61f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:23.338 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:23.352+0000 7fce3b61f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:23.339 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:23.352+0000 7fce3b61f4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:50:23.339 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:23.352+0000 7fce3b61f4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:50:23.339 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:23.352+0000 7fce3b61f4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:50:23.412 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:23.426+0000 7fce3b61f4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:50:29.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:50:29.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:50:29.141 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:50:29.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:50:29.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:50:29.487 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:50:29.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:50:29.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:50:29.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:50:29.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:50:29.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:50:29.513 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:29.527+0000 7f2fca00e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:29.515 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:29.529+0000 7f2fca00e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:29.516 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:29.530+0000 7f2fca00e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:50:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:50:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:50:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:50:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:50:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:50:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:50:29.757 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:50:29.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:29.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:30.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:30.038 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:30.052+0000 7f2fca00e4c0 -1 Falling back to public interface 2024-02-05T05:50:31.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:31.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:31.025 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:50:31.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:50:31.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:31.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:31.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:32.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:32.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:32.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:50:32.304 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:50:32.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:32.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:32.410 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:50:32.424+0000 7f2fca00e4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:50:32.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:33.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:33.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:33.637 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:50:33.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:50:33.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:33.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:33.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:34.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:34.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:34.920 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:50:34.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:50:34.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:34.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:35.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:36.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:36.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:36.199 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:50:36.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:50:36.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:36.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:36.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:37.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:37.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:37.477 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:50:37.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:50:37.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:37.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:37.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:38.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:38.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:38.756 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:50:38.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:50:38.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:38.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:39.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:40.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:40.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:40.123 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:50:40.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:50:40.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:40.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:40.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:41.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:41.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:41.431 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:50:41.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:50:41.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:41.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:41.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:42.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:42.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:42.709 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:50:42.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:50:42.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:42.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:43.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:44.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:44.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:44.023 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:50:44.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:50:44.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:44.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:44.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:45.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:45.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:45.337 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:50:45.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:50:45.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:45.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:45.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:46.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:46.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:46.949 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:50:46.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:50:46.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:46.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:47.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:48.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:48.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:48.408 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:50:48.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:50:48.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:48.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:48.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:49.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:49.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:49.860 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:50:49.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:50:49.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:49.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:50.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:51.331 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:50:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:50:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:51.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:52.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:52.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:52.638 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:50:52.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:50:52.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:52.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:53.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:54.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:54.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:54.343 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:50:54.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:50:54.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:54.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:55.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:50:56.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:50:56.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:50:56.141 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:50:56.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:50:56.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:50:56.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:50:56.415 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1509908004,v1:127.0.0.1:6819/1509908004] [v2:127.0.0.1:6820/1509908004,v1:127.0.0.1:6821/1509908004] exists,up 880dce46-8ffb-4945-a247-948346d2bc8e 2024-02-05T05:50:56.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:50:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:50:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:50:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-02-05T05:50:56.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:50:56.729 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:50:56.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:50:56.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:50:57.140 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:50:57.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:50:58.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:50:59.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-02-05T05:50:59.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:50:59.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:50:59.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:50:59.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:50:59.510 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:50:59.510 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:50:59.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:50:59.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:50:59.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:50:59.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:50:59.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:50:59.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:50:59.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:50:59.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:50:59.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:50:59.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:50:59.877 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:50:59.877 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:50:59.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:50:59.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:59.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:50:59.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T05:50:59.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T05:50:59.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T05:50:59.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:50:59.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:51:00.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-02-05T05:51:00.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-02-05T05:51:00.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-02-05T05:51:00.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:00.084 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:51:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-02-05T05:51:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-02-05T05:51:00.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509449' 2024-02-05T05:51:00.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:00.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T05:51:00.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:00.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:51:00.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T05:51:00.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:00.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T05:51:00.191 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836500 2024-02-05T05:51:00.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T05:51:00.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:00.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836500 2024-02-05T05:51:00.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:51:01.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:51:01.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:01.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-02-05T05:51:01.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:01.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-02-05T05:51:01.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:01.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:51:01.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-02-05T05:51:01.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:01.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-02-05T05:51:01.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-02-05T05:51:01.735 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672974 2024-02-05T05:51:01.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:51:02.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672974 2024-02-05T05:51:02.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:02.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-02-05T05:51:02.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:02.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:51:02.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-02-05T05:51:02.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:02.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-02-05T05:51:02.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-02-05T05:51:02.006 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509449 2024-02-05T05:51:02.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:02.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-02-05T05:51:02.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:51:02.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:02.282 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:02.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:51:02.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:51:02.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:51:02.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:51:02.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:51:02.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:51:02.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:51:02.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:51:02.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:51:02.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:51:02.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:02.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:03.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:51:03.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:51:03.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:51:03.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-02-05T05:51:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T05:51:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:51:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T05:51:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:51:03.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T05:51:03.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T05:51:03.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T05:51:04.252 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T05:51:04.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:51:05.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T05:51:05.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T05:51:05.747 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 2 allow_ec_overwrites to true 2024-02-05T05:51:05.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:51:05.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:51:05.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:51:05.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:51:05.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:51:05.757 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:51:05.757 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:51:05.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:51:05.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:51:05.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:51:05.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:51:05.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:51:05.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:51:05.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:51:05.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:51:05.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:51:06.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:51:06.156 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:51:06.156 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:51:06.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:51:06.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:06.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:51:06.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-02-05T05:51:06.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-02-05T05:51:06.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-02-05T05:51:06.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:06.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:51:06.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-02-05T05:51:06.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-02-05T05:51:06.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-02-05T05:51:06.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:06.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:51:06.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T05:51:06.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T05:51:06.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509452' 2024-02-05T05:51:06.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:06.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-02-05T05:51:06.467 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:06.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:51:06.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-02-05T05:51:06.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:06.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-02-05T05:51:06.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-02-05T05:51:06.470 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836502 2024-02-05T05:51:06.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:06.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-02-05T05:51:06.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:06.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-02-05T05:51:06.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:06.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:51:06.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-02-05T05:51:06.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:06.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-02-05T05:51:06.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-02-05T05:51:06.739 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672976 2024-02-05T05:51:06.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:51:07.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-02-05T05:51:07.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:07.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T05:51:07.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:07.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:51:07.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T05:51:07.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:07.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T05:51:07.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T05:51:07.014 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509452 2024-02-05T05:51:07.014 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:07.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T05:51:07.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:51:07.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:07.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:07.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:51:07.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:51:07.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:51:07.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:51:07.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:51:07.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:51:07.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:51:07.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:51:07.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:51:07.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:51:07.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:07.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:51:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:51:08.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:51:08.609 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:51:08.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:51:08.986 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:51:08.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:51:08.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:51:08.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:51:09.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-02-05T05:51:09.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:51:09.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-02-05T05:51:09.224 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-02-05T05:51:09.224 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:51:09.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:51:09.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:51:09.839 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:51:09.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:51:10.342 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:51:10.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:51:10.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:51:10.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:51:10.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-02-05T05:51:10.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T05:51:10.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-02-05T05:51:10.622 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-02-05T05:51:10.622 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T05:51:10.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-02-05T05:51:10.899 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:51:10.899 INFO:tasks.workunit.client.0.smithi188.stderr:0' 2024-02-05T05:51:10.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-02-05T05:51:10.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-02-05T05:51:10.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-02-05T05:51:10.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-02-05T05:51:10.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:51:10.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:51:10.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:10.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:10.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:10.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:10.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:11.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:11.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:51:11.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:51:11.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:11.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:11.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:11.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:11.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:12.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:12.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:51:12.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T05:51:12.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:12.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:12.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:12.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:12.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:12.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:12.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-02-05T05:51:12.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-02-05T05:51:13.911 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-02-05T05:51:14.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-02-05T05:51:14.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-02-05T05:51:15.565 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-02-05T05:51:16.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-02-05T05:51:16.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-02-05T05:51:16.993 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-02-05T05:51:17.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-02-05T05:51:17.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-02-05T05:51:18.588 INFO:tasks.workunit.client.0.smithi188.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-02-05T05:51:19.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-02-05T05:51:19.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:51:19.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:19.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:19.116 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:51:19.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:51:19.118 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:51:19.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:51:19.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:51:19.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:51:19.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:51:19.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:51:19.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:51:19.143 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:19.158+0000 7fec6f6654c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:19.145 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:19.160+0000 7fec6f6654c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:19.147 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:19.161+0000 7fec6f6654c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:19.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:51:19.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:19.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:51:19.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:19.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:19.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:19.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:19.394 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:51:19.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:19.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:19.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:19.669 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:19.684+0000 7fec6f6654c0 -1 Falling back to public interface 2024-02-05T05:51:20.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:20.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:20.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:20.662 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:51:20.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:20.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:20.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:21.935 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:51:21.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:21.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:21.964 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:21.978+0000 7fec6f6654c0 -1 osd.0 33 log_to_monitors true 2024-02-05T05:51:22.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:23.217 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:51:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:23.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:23.491 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 34 last_clean_interval [5,33) [v2:127.0.0.1:6802/332352184,v1:127.0.0.1:6803/332352184] [v2:127.0.0.1:6804/332352184,v1:127.0.0.1:6805/332352184] exists,up 507968b6-910b-45e1-945e-948771d82a01 2024-02-05T05:51:23.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:51:23.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:51:23.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:23.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:51:23.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:51:23.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:51:23.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:51:23.494 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.1 2024-02-05T05:51:23.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:51:23.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:51:23.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:51:23.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:51:23.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:51:23.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:51:23.516 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:23.531+0000 7fce01e184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:23.520 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:23.533+0000 7fce01e184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:23.520 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:23.534+0000 7fce01e184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:23.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:51:24.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:24.042 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:24.057+0000 7fce01e184c0 -1 Falling back to public interface 2024-02-05T05:51:25.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:25.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:25.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:25.025 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:51:25.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:25.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:51:25.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:26.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:26.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:26.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:26.302 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:51:26.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:26.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:51:26.349 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:26.364+0000 7fce01e184c0 -1 osd.1 35 log_to_monitors true 2024-02-05T05:51:26.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:27.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:27.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:27.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:27.583 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:51:27.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:27.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:51:27.864 INFO:tasks.workunit.client.0.smithi188.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1847895841,v1:127.0.0.1:6811/1847895841] [v2:127.0.0.1:6812/1847895841,v1:127.0.0.1:6813/1847895841] exists,up ea5cb4fc-03f2-4640-833f-2940ac9bd8bd 2024-02-05T05:51:27.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:51:27.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:51:27.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:51:27.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:27.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:27.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.2 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T05:51:27.867 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T05:51:27.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T05:51:27.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:51:27.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:51:27.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:51:27.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:27.904+0000 7f90b320d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:27.891 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:27.906+0000 7f90b320d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:27.892 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:27.907+0000 7f90b320d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T05:51:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:51:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:28.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:28.137 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:51:28.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:28.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:51:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:28.419 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:28.434+0000 7f90b320d4c0 -1 Falling back to public interface 2024-02-05T05:51:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:51:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:29.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:51:29.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:30.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:30.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:30.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:30.676 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:51:30.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:30.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:51:30.914 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:30.929+0000 7f90b320d4c0 -1 osd.2 37 log_to_monitors true 2024-02-05T05:51:30.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:31.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:31.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:31.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:31.995 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:51:31.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:31.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:51:32.487 INFO:tasks.workunit.client.0.smithi188.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/3314888993,v1:127.0.0.1:6819/3314888993] [v2:127.0.0.1:6820/3314888993,v1:127.0.0.1:6821/3314888993] exists,up 880dce46-8ffb-4945-a247-948346d2bc8e 2024-02-05T05:51:32.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:51:32.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:51:32.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:51:32.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-02-05T05:51:32.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:51:32.852 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' removed 2024-02-05T05:51:32.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:51:32.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:51:33.330 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:51:33.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:51:34.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:51:35.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-02-05T05:51:35.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:51:35.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:51:35.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:51:35.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:51:35.565 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:51:35.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:51:35.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:51:35.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:51:35.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:51:35.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:51:35.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:51:35.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:51:35.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:51:35.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:51:35.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:51:35.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:51:35.929 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:51:35.929 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:51:35.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:51:35.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:35.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:51:36.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691844 2024-02-05T05:51:36.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691844 2024-02-05T05:51:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844' 2024-02-05T05:51:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:36.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:51:36.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593731 2024-02-05T05:51:36.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593731 2024-02-05T05:51:36.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731' 2024-02-05T05:51:36.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:51:36.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:51:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-02-05T05:51:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-02-05T05:51:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593731 2-201863462915' 2024-02-05T05:51:36.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:36.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691844 2024-02-05T05:51:36.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:36.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:51:36.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691844 2024-02-05T05:51:36.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:36.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691844 2024-02-05T05:51:36.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2024-02-05T05:51:36.242 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 171798691844 2024-02-05T05:51:36.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:36.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-02-05T05:51:36.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:51:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:51:37.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:51:37.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691844 -lt 171798691844 2024-02-05T05:51:37.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:37.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593731 2024-02-05T05:51:37.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:37.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:51:37.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593731 2024-02-05T05:51:37.788 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:37.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593731 2024-02-05T05:51:37.789 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 184683593731 2024-02-05T05:51:37.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-02-05T05:51:37.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:51:38.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593731 2024-02-05T05:51:38.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:51:38.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-02-05T05:51:38.125 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:51:38.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:51:38.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-02-05T05:51:38.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:51:38.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-02-05T05:51:38.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-02-05T05:51:38.128 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 201863462915 2024-02-05T05:51:38.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:51:38.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-02-05T05:51:38.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:51:38.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:38.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:38.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:51:38.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:51:38.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:51:38.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:51:38.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:51:38.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:51:38.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:51:38.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:51:38.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:51:38.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:51:38.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:51:38.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:51:39.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:51:39.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:51:39.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:51:39.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-02-05T05:51:39.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T05:51:39.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-02-05T05:51:39.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-02-05T05:51:39.325 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:51:39.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-02-05T05:51:39.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-02-05T05:51:39.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-02-05T05:51:39.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-02-05T05:51:39.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:39.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:39.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:39.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:39.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:51:04.150130+0000 2024-02-05T05:51:39.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-02-05T05:51:40.108 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-02-05T05:51:40.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-02-05T05:51:04.150130+0000 2024-02-05T05:51:40.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:51:40.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:51:04.150130+0000 2024-02-05T05:51:40.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:51:40.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:51:40.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:40.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:40.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:40.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:40.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:40.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:40.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:04.150130+0000 '>' 2024-02-05T05:51:04.150130+0000 2024-02-05T05:51:40.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:41.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:41.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:41.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:41.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:41.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:41.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:41.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:41.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:04.150130+0000 '>' 2024-02-05T05:51:04.150130+0000 2024-02-05T05:51:41.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:51:42.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:51:42.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:51:42.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:51:42.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:51:42.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:51:42.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:51:42.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:51:42.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:51:41.092211+0000 '>' 2024-02-05T05:51:04.150130+0000 2024-02-05T05:51:42.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:51:42.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-02-05T05:51:42.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-02-05T05:51:42.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-02-05T05:51:42.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-02-05T05:51:42.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-02-05T05:51:42.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-02-05T05:51:43.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:43.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:43.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:43.157 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:51:43.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:51:43.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:51:43.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:51:43.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:51:43.160 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:51:43.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:51:43.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:51:43.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:51:43.161 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:51:43.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:51:43.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:51:43.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:51:43.179 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:51:43.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:43.180 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:43.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:51:43.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:51:43.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:51:43.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:51:43.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:51:43.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:51:43.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:51:43.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:51:43.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:51:43.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:51:43.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:51:43.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:51:43.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:51:43.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:51:43.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:51:43.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:51:43.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:51:43.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:51:43.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:51:43.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:51:43.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:43.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:43.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:51:43.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:51:43.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:51:43.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:51:43.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:51:43.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:43.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:43.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:51:43.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:51:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:51:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:51:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:51:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-02-05T05:51:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-02-05T05:51:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:51:43.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T05:51:43.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:51:43.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:43.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:43.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:43.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:43.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:43.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:43.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T05:51:43.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:51:43.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:51:43.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:51:43.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:51:43.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:51:43.751 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:51:43.752 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:51:43.752 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:51:43.752 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:51:43.752 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:51:43.752 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:43.752 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:43.752 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:51:43.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:51:43.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:51:43.831 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:51:43.832 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:51:43.832 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:43.832 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:43.832 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:51:43.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:51:43.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:51:43.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-02-05T05:51:43.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:51:43.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:51:43.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:51:43.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:51:43.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:51:43.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:51:44.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:51:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:44.720 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:44.720 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:44.720 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:44.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:44.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:51:44.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:51:44.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:51:44.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:51:44.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:51:44.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30042066-7a8d-42fb-8f40-027516763993 2024-02-05T05:51:44.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 30042066-7a8d-42fb-8f40-027516763993' 2024-02-05T05:51:44.754 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 30042066-7a8d-42fb-8f40-027516763993 2024-02-05T05:51:44.754 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:51:44.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwd8BlWpXALhAAnedxWJupQCF0bv9fPIC2ZQ== 2024-02-05T05:51:44.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwd8BlWpXALhAAnedxWJupQCF0bv9fPIC2ZQ=="}' 2024-02-05T05:51:44.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30042066-7a8d-42fb-8f40-027516763993 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:51:44.975 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:51:44.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:51:44.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQBwd8BlWpXALhAAnedxWJupQCF0bv9fPIC2ZQ== --osd-uuid 30042066-7a8d-42fb-8f40-027516763993 2024-02-05T05:51:45.017 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:45.032+0000 7f0f9e60c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:45.026 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:45.041+0000 7f0f9e60c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:45.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:45.042+0000 7f0f9e60c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:45.027 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:45.043+0000 7f0f9e60c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:51:45.028 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:45.043+0000 7f0f9e60c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:51:45.028 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:45.043+0000 7f0f9e60c4c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:51:45.118 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:45.133+0000 7f0f9e60c4c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:51:50.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:51:50.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:51:50.586 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:51:50.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:51:50.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:51:50.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:51:50.934 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:51:50.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:51:50.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:51:50.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:51:50.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:51:50.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:50.977+0000 7f9a230114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:50.976 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:50.991+0000 7f9a230114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:50.977 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:50.992+0000 7f9a230114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:51:51.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:51:51.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:51:51.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:51:51.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:51:51.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:51:51.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:51.236 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:51:51.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:51:51.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:51.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:51.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:51.515 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:51.531+0000 7f9a230114c0 -1 Falling back to public interface 2024-02-05T05:51:52.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:52.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:52.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:51:52.501 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:51:52.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:52.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:52.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:53.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:53.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:53.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:51:53.779 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:51:53.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:53.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:53.922 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:51:53.937+0000 7f9a230114c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:51:54.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:55.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:55.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:55.107 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:51:55.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:51:55.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:55.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:55.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:56.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:56.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:56.380 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:51:56.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:51:56.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:56.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:56.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:57.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:57.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:57.696 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:51:57.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:51:57.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:57.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:57.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:51:58.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:51:58.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:51:58.980 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:51:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:51:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:51:58.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:51:59.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:00.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:00.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:00.262 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:52:00.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:52:00.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:00.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:00.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:01.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:01.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:01.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:52:01.558 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:52:01.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:01.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:01.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:02.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:02.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:02.938 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:52:02.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:52:02.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:02.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:03.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:04.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:04.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:04.217 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:52:04.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:52:04.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:04.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:04.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:05.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:05.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:05.532 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:52:05.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:52:05.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:05.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:05.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:06.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:06.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:06.808 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:52:06.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:52:06.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:06.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:07.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:08.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:08.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:08.122 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:52:08.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:52:08.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:08.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:08.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:09.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:09.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:09.588 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:52:09.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:52:09.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:09.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:10.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:11.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:11.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:11.319 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:52:11.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:52:11.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:11.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:11.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:12.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:12.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:12.932 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:52:12.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:52:12.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:12.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:13.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:14.791 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:52:14.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:52:14.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:14.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:15.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:16.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:16.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:16.569 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:52:16.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:52:16.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:16.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:52:16.847 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/137542202,v1:127.0.0.1:6803/137542202] [v2:127.0.0.1:6804/137542202,v1:127.0.0.1:6805/137542202] exists,up 30042066-7a8d-42fb-8f40-027516763993 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:52:16.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:52:16.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:52:16.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9806d283-e213-4f63-a0d7-24e4da00ee98 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9806d283-e213-4f63-a0d7-24e4da00ee98' 2024-02-05T05:52:16.855 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 9806d283-e213-4f63-a0d7-24e4da00ee98 2024-02-05T05:52:16.856 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:52:16.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQd8Bl3G64NBAAgBUe4TNSJLJYfcGpSYLemg== 2024-02-05T05:52:16.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQd8Bl3G64NBAAgBUe4TNSJLJYfcGpSYLemg=="}' 2024-02-05T05:52:16.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9806d283-e213-4f63-a0d7-24e4da00ee98 -i td/osd-scrub-repair/1/new.json 2024-02-05T05:52:17.271 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:52:17.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:52:17.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQCQd8Bl3G64NBAAgBUe4TNSJLJYfcGpSYLemg== --osd-uuid 9806d283-e213-4f63-a0d7-24e4da00ee98 2024-02-05T05:52:17.313 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:17.329+0000 7f3089fa94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:17.364 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:17.380+0000 7f3089fa94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:17.365 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:17.381+0000 7f3089fa94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:17.365 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:17.381+0000 7f3089fa94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:52:17.365 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:17.381+0000 7f3089fa94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:52:17.366 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:17.381+0000 7f3089fa94c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:52:17.439 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:17.455+0000 7f3089fa94c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:52:23.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:52:23.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:52:23.539 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:52:23.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:52:23.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:52:23.886 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:52:23.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:52:23.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:52:23.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:52:23.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:52:23.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:52:23.914 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:23.930+0000 7f4d608144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:23.916 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:23.932+0000 7f4d608144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:23.917 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:23.933+0000 7f4d608144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:52:24.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:52:24.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:52:24.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:52:24.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:52:24.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:52:24.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:24.153 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:52:24.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:52:24.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:24.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:24.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:24.444 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:24.460+0000 7f4d608144c0 -1 Falling back to public interface 2024-02-05T05:52:25.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:25.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:25.421 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:52:25.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:52:25.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:25.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:25.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:26.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:26.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:26.695 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:52:26.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:52:26.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:26.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:26.804 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:52:26.820+0000 7f4d608144c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:52:26.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:27.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:27.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:27.989 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:52:27.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:52:27.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:27.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:28.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:29.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:29.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:29.268 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:52:29.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:52:29.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:29.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:29.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:30.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:30.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:30.547 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:52:30.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:52:30.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:30.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:30.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:31.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:31.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:31.927 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:52:31.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:52:31.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:31.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:32.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:33.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:33.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:33.209 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:52:33.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:52:33.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:33.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:33.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:34.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:34.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:34.511 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:52:34.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:52:34.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:34.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:34.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:35.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:35.911 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:52:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:52:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:35.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:36.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:37.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:37.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:37.185 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:52:37.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:52:37.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:37.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:37.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:38.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:38.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:38.535 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:52:38.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:52:38.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:38.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:38.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:39.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:39.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:39.965 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:52:39.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:52:39.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:39.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:40.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:41.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:41.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:41.536 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:52:41.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:52:41.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:41.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:42.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:43.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:43.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:43.050 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:52:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:52:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:43.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:43.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:44.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:44.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:44.910 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:52:44.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:52:44.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:44.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:45.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:46.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:46.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:46.722 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:52:46.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:52:46.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:46.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:47.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:48.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:48.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:48.458 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:52:48.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:52:48.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:48.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:49.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:52:50.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:52:50.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:52:50.361 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:52:50.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:52:50.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:52:50.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:52:50.639 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1353165593,v1:127.0.0.1:6811/1353165593] [v2:127.0.0.1:6812/1353165593,v1:127.0.0.1:6813/1353165593] exists,up 9806d283-e213-4f63-a0d7-24e4da00ee98 2024-02-05T05:52:50.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:52:50.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:52:50.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:52:50.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-02-05T05:52:50.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T05:52:50.922 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T05:52:50.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T05:52:50.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T05:52:51.306 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T05:52:51.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:52:52.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T05:52:53.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-02-05T05:52:53.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:53.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:53.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:53.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:53.911 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:53.911 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:53.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:53.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:53.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:54.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:54.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:54.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:54.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:54.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:54.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:54.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:54.276 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:52:54.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:54.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:54.276 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-02-05T05:52:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-02-05T05:52:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-02-05T05:52:54.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:54.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:54.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T05:52:54.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T05:52:54.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672964' 2024-02-05T05:52:54.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:54.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-02-05T05:52:54.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:54.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:54.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-02-05T05:52:54.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:54.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-02-05T05:52:54.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-02-05T05:52:54.486 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836491 2024-02-05T05:52:54.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:54.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-02-05T05:52:54.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:52:55.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:52:55.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:56.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-02-05T05:52:56.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:56.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T05:52:56.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:56.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:52:56.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T05:52:56.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:56.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T05:52:56.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T05:52:56.049 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672964 2024-02-05T05:52:56.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:52:56.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-02-05T05:52:56.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:52:56.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:56.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:56.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T05:52:56.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:52:56.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:52:56.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:52:56.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:52:56.650 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:52:56.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:52:56.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:52:56.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T05:52:56.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:52:56.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:52:56.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:52:57.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T05:52:57.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:52:57.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:52:57.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-02-05T05:52:57.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-02-05T05:52:57.605 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool' created 2024-02-05T05:52:57.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:52:58.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-02-05T05:52:58.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:52:58.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:52:58.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:52:58.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:52:58.619 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:52:58.619 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:52:58.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:52:58.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:52:58.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:52:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:52:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:52:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:52:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:52:58.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:52:58.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:52:59.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:52:59.041 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:52:59.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:52:59.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:59.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:52:59.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-02-05T05:52:59.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-02-05T05:52:59.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-02-05T05:52:59.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:52:59.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:52:59.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-02-05T05:52:59.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-02-05T05:52:59.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672966' 2024-02-05T05:52:59.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:52:59.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-02-05T05:52:59.268 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:52:59.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:52:59.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-02-05T05:52:59.270 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:52:59.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-02-05T05:52:59.271 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836493 2024-02-05T05:52:59.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-02-05T05:52:59.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:52:59.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-02-05T05:52:59.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:53:00.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:53:00.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:00.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-02-05T05:53:00.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:00.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-02-05T05:53:00.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:00.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:53:00.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-02-05T05:53:00.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:00.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-02-05T05:53:00.817 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672966 2024-02-05T05:53:00.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-02-05T05:53:00.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:53:01.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-02-05T05:53:01.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:53:01.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:53:01.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:53:01.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:53:01.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:53:01.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:53:01.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:53:01.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:53:01.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:53:01.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:53:01.438 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:53:01.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:53:01.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:53:01.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:53:01.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:53:02.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:53:02.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:53:02.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:53:02.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-02-05T05:53:02.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-02-05T05:53:02.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:53:02.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-02-05T05:53:02.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-02-05T05:53:02.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-02-05T05:53:02.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-02-05T05:53:02.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-02-05T05:53:02.384 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is unset 2024-02-05T05:53:02.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-02-05T05:53:02.945 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is unset 2024-02-05T05:53:02.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:53:02.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:53:02.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-02-05T05:53:03.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-02-05T05:53:03.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-02-05T05:53:03.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-02-05T05:53:03.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-02-05T05:53:03.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:53:03.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-02-05T05:53:03.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-02-05T05:53:03.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-02-05T05:53:03.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-02-05T05:53:03.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-02-05T05:53:03.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:53:03.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T05:53:03.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:53:03.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:53:03.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:53:03.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:53:03.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:53:03.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:53:04.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:53:04.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:53:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:53:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:53:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:53:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:53:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:53:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-02-05T05:53:06.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:53:06.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:53:06.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:53:06.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:53:06.132 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T05:53:06.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:53:06.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:53:06.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:53:06.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:53:06.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:53:06.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:53:06.160 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:06.176+0000 7fc6924b64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:06.162 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:06.179+0000 7fc6924b64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:06.163 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:06.180+0000 7fc6924b64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:06.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:53:06.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:53:06.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:53:06.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:53:06.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:53:06.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:06.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:53:06.408 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T05:53:06.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:06.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:06.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:06.681 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:06.698+0000 7fc6924b64c0 -1 Falling back to public interface 2024-02-05T05:53:07.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:07.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:07.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:53:07.676 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:53:07.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:07.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:07.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:08.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:08.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:08.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:53:08.955 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T05:53:08.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:08.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:09.189 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:09.205+0000 7fc6924b64c0 -1 osd.0 24 log_to_monitors true 2024-02-05T05:53:09.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:10.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:10.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:10.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:53:10.234 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T05:53:10.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:10.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/544310684,v1:127.0.0.1:6803/544310684] [v2:127.0.0.1:6804/544310684,v1:127.0.0.1:6805/544310684] exists,up 30042066-7a8d-42fb-8f40-027516763993 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:53:10.512 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:53:10.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:53:10.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:53:10.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:53:10.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:53:10.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:53:10.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:53:10.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:53:10.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:53:10.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:53:10.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:53:10.886 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:53:10.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:53:10.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:10.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:53:10.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084290 2024-02-05T05:53:10.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084290 2024-02-05T05:53:10.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290' 2024-02-05T05:53:10.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:10.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:53:11.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-02-05T05:53:11.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-02-05T05:53:11.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084290 1-42949672970' 2024-02-05T05:53:11.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:11.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084290 2024-02-05T05:53:11.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:11.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:53:11.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084290 2024-02-05T05:53:11.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:11.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084290 2024-02-05T05:53:11.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2024-02-05T05:53:11.139 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 120259084290 2024-02-05T05:53:11.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:11.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 120259084290 2024-02-05T05:53:11.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:53:12.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:53:12.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:12.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 120259084290 2024-02-05T05:53:12.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:53:13.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:53:13.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:13.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084290 -lt 120259084290 2024-02-05T05:53:13.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:13.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-02-05T05:53:13.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:13.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:53:13.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-02-05T05:53:13.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:13.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-02-05T05:53:13.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-02-05T05:53:13.960 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672970 2024-02-05T05:53:13.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:53:14.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-02-05T05:53:14.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:53:14.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:53:14.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:53:14.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T05:53:14.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:53:14.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:53:14.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:53:14.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:53:14.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:53:14.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:53:14.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:53:14.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T05:53:14.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:53:14.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:53:14.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:53:15.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T05:53:15.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:53:15.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:53:15.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-02-05T05:53:15.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T05:53:15.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-02-05T05:53:15.217 INFO:tasks.workunit.client.0.smithi188.stderr:No scrub information available for pg 2.0 2024-02-05T05:53:15.217 INFO:tasks.workunit.client.0.smithi188.stderr:error 2: (2) No such file or directory 2024-02-05T05:53:15.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-02-05T05:53:15.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-02-05T05:53:15.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-02-05T05:53:15.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T05:53:15.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:15.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:53:15.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:15.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T05:53:15.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-02-05T05:52:57.575905+0000 2024-02-05T05:53:15.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-02-05T05:53:15.769 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-02-05T05:53:15.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-02-05T05:52:57.575905+0000 last_deep_scrub_stamp 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:52:57.575905+0000 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:15.782 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T05:53:16.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:52:57.575905+0000 '>' 2024-02-05T05:52:57.575905+0000 2024-02-05T05:53:16.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:53:17.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:53:17.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:17.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T05:53:17.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:17.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:53:17.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:17.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T05:53:17.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:52:57.575905+0000 '>' 2024-02-05T05:52:57.575905+0000 2024-02-05T05:53:17.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:53:18.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:53:18.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:18.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T05:53:18.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:18.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:53:18.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:18.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T05:53:18.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:52:57.575905+0000 '>' 2024-02-05T05:52:57.575905+0000 2024-02-05T05:53:18.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:53:19.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:53:19.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:19.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T05:53:19.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:19.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:53:19.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:19.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T05:53:19.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:52:57.575905+0000 '>' 2024-02-05T05:52:57.575905+0000 2024-02-05T05:53:19.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:53:20.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:53:20.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-02-05T05:53:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-02-05T05:53:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:20.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-02-05T05:53:21.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:53:16.206136+0000 '>' 2024-02-05T05:52:57.575905+0000 2024-02-05T05:53:21.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:53:21.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T05:53:21.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-02-05T05:53:21.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-02-05T05:53:21.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-02-05T05:53:21.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:53:21.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:53:21.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:53:21.472 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:53:21.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:53:21.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:21.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:53:21.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084293 2024-02-05T05:53:21.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084293 2024-02-05T05:53:21.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084293' 2024-02-05T05:53:21.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:21.574 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:53:21.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-02-05T05:53:21.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-02-05T05:53:21.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084293 1-42949672973' 2024-02-05T05:53:21.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:21.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084293 2024-02-05T05:53:21.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:21.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:53:21.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084293 2024-02-05T05:53:21.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:21.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084293 2024-02-05T05:53:21.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084293' 2024-02-05T05:53:21.682 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 120259084293 2024-02-05T05:53:21.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:21.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084292 -lt 120259084293 2024-02-05T05:53:21.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:53:22.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:53:22.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:23.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084293 -lt 120259084293 2024-02-05T05:53:23.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:23.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-02-05T05:53:23.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:23.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:53:23.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-02-05T05:53:23.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:23.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-02-05T05:53:23.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-02-05T05:53:23.231 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672973 2024-02-05T05:53:23.231 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:53:23.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-02-05T05:53:23.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-02-05T05:53:23.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:23.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:53:23.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:23.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:53:23.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-02-05T05:53:16.206136+0000 2024-02-05T05:53:23.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-02-05T05:53:23.861 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:53:23.862 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:53:23.862 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:53:23.862 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:51:43.879407+0000" 2024-02-05T05:53:23.862 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:53:23.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-02-05T05:53:16.206136+0000 2024-02-05T05:53:23.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:53:16.206136+0000 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:23.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:53:24.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:53:16.206136+0000 '>' 2024-02-05T05:53:16.206136+0000 2024-02-05T05:53:24.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:53:25.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:53:25.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:25.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:53:25.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:25.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:53:25.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:25.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:53:25.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:53:24.204385+0000 '>' 2024-02-05T05:53:16.206136+0000 2024-02-05T05:53:25.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:53:25.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-02-05T05:53:25.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T05:53:25.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-02-05T05:53:25.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-02-05T05:53:25.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-02-05T05:53:25.800 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:53:25.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-02-05T05:53:25.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-02-05T05:53:25.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:53:25.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:53:26.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:53:26.184 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:53:26.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:53:26.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:26.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:53:26.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084295 2024-02-05T05:53:26.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084295 2024-02-05T05:53:26.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295' 2024-02-05T05:53:26.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:26.288 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:53:26.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-02-05T05:53:26.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-02-05T05:53:26.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084295 1-42949672975' 2024-02-05T05:53:26.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:26.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084295 2024-02-05T05:53:26.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:26.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:53:26.393 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084295 2024-02-05T05:53:26.393 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:26.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084295 2024-02-05T05:53:26.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2024-02-05T05:53:26.394 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 120259084295 2024-02-05T05:53:26.394 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:26.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084294 -lt 120259084295 2024-02-05T05:53:26.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:53:27.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:53:27.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:27.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084295 -lt 120259084295 2024-02-05T05:53:27.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:27.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-02-05T05:53:27.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:27.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:53:27.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-02-05T05:53:27.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:27.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-02-05T05:53:27.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-02-05T05:53:27.938 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672975 2024-02-05T05:53:27.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:53:28.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-02-05T05:53:28.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-02-05T05:53:33.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-02-05T05:53:33.313 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:53:33.313 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:53:33.313 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:53:33.313 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:51:53.331280+0000" 2024-02-05T05:53:33.313 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:53:33.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-02-05T05:53:33.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-02-05T05:53:33.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-02-05T05:53:33.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-02-05T05:53:34.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-02-05T05:53:34.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-02-05T05:53:34.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-02-05T05:53:34.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-02-05T05:53:34.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T05:53:34.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-02-05T05:53:34.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-02-05T05:53:34.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-02-05T05:53:34.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:53:34.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:53:34.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:53:34.629 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:53:34.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:53:34.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:34.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:53:34.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084298 2024-02-05T05:53:34.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084298 2024-02-05T05:53:34.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084298' 2024-02-05T05:53:34.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:53:34.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:53:34.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-02-05T05:53:34.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-02-05T05:53:34.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-120259084298 1-42949672977' 2024-02-05T05:53:34.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:34.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-120259084298 2024-02-05T05:53:34.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:34.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:53:34.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-120259084298 2024-02-05T05:53:34.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:34.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084298 2024-02-05T05:53:34.840 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 120259084298 2024-02-05T05:53:34.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 120259084298' 2024-02-05T05:53:34.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:35.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084297 -lt 120259084298 2024-02-05T05:53:35.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:53:36.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:53:36.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:36.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084297 -lt 120259084298 2024-02-05T05:53:36.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:53:37.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:53:37.384 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:53:37.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084298 -lt 120259084298 2024-02-05T05:53:37.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:53:37.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-02-05T05:53:37.659 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:53:37.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:53:37.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-02-05T05:53:37.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:53:37.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-02-05T05:53:37.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-02-05T05:53:37.663 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672977 2024-02-05T05:53:37.663 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:53:37.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-02-05T05:53:37.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-02-05T05:53:37.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-02-05T05:53:37.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-02-05T05:53:37.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:37.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:53:37.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:37.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:53:38.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-01-29T05:51:53.331280+0000 2024-02-05T05:53:38.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-02-05T05:53:38.448 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 2.0 on osd.0 to scrub 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-01-29T05:51:53.331280+0000 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-01-29T05:51:53.331280+0000 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:38.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:53:38.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T05:51:53.331280+0000 '>' 2024-01-29T05:51:53.331280+0000 2024-02-05T05:53:38.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:53:39.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:53:39.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:39.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:53:39.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:39.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:53:39.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:39.725 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:53:39.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T05:51:53.331280+0000 '>' 2024-01-29T05:51:53.331280+0000 2024-02-05T05:53:39.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:53:40.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:53:40.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:53:40.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-02-05T05:53:40.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-02-05T05:53:40.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:53:40.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:53:40.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-02-05T05:53:41.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:53:39.302131+0000 '>' 2024-01-29T05:51:53.331280+0000 2024-02-05T05:53:41.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:53:41.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-02-05T05:53:41.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-02-05T05:53:41.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-02-05T05:53:41.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-02-05T05:53:41.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:53:41.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:53:41.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:53:41.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:53:41.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:53:41.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:53:41.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:53:41.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:53:41.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:53:41.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:53:41.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:53:41.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:53:41.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:53:41.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:53:41.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:53:41.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:53:41.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:53:41.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:53:41.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:53:41.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:53:41.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:53:41.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:53:41.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:53:41.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:53:41.430 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:41.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:41.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:53:41.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:53:41.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:53:41.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:53:41.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:53:41.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:53:41.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:53:41.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:53:41.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:53:41.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:53:41.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:53:41.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:53:41.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:53:41.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:53:41.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:53:41.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:53:41.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:53:41.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:53:41.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:53:41.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:53:41.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:41.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:41.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:53:41.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:53:41.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:53:41.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:53:41.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:53:41.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:41.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:41.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:53:41.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:53:41.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:53:41.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:53:41.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:53:41.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:53:41.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:53:41.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:53:41.810 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:53:41.810 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:41.810 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:41.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:53:41.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:53:41.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:53:41.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:53:41.995 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:53:41.996 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:41.996 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:41.996 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:53:41.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:53:41.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:53:42.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:53:42.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:53:42.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:53:42.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:53:42.075 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:53:42.075 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:53:42.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:53:42.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-02-05T05:53:42.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:53:42.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:53:42.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:53:42.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:53:42.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:53:42.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:53:42.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:53:42.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:53:42.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:53:42.962 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:53:42.962 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:42.962 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:42.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:53:42.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:53:42.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:53:42.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:53:42.990 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-02-05T05:53:42.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-02-05T05:53:42.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:53:42.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:53:42.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:53:42.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:53:42.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=12e33266-2d9a-4dc8-b7b4-c1b67ae2f8d6 2024-02-05T05:53:42.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 12e33266-2d9a-4dc8-b7b4-c1b67ae2f8d6' 2024-02-05T05:53:42.999 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 12e33266-2d9a-4dc8-b7b4-c1b67ae2f8d6 2024-02-05T05:53:42.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:53:43.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnd8Bl1LrhARAAuGKsKHizJw1ru+ix4cXlVg== 2024-02-05T05:53:43.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnd8Bl1LrhARAAuGKsKHizJw1ru+ix4cXlVg=="}' 2024-02-05T05:53:43.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 12e33266-2d9a-4dc8-b7b4-c1b67ae2f8d6 -i td/osd-scrub-repair/0/new.json 2024-02-05T05:53:43.218 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:53:43.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:53:43.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDnd8Bl1LrhARAAuGKsKHizJw1ru+ix4cXlVg== --osd-uuid 12e33266-2d9a-4dc8-b7b4-c1b67ae2f8d6 2024-02-05T05:53:43.258 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:43.275+0000 7f4852d724c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:43.268 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:43.286+0000 7f4852d724c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:43.269 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:43.287+0000 7f4852d724c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:43.270 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:43.287+0000 7f4852d724c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:53:43.270 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:43.287+0000 7f4852d724c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:53:43.270 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:43.287+0000 7f4852d724c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:53:43.361 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:43.378+0000 7f4852d724c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:53:48.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:53:48.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:53:48.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:53:48.898 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:53:48.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:53:49.246 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:53:49.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:53:49.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:53:49.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:53:49.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:53:49.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:53:49.274 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:49.291+0000 7f3cda80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:49.276 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:49.293+0000 7f3cda80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:49.277 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:49.294+0000 7f3cda80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:53:49.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:53:49.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:53:49.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:53:49.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:53:49.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:53:49.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:49.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:53:49.518 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:53:49.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:49.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:49.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:49.800 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:49.818+0000 7f3cda80e4c0 -1 Falling back to public interface 2024-02-05T05:53:50.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:50.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:50.783 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:53:50.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:53:50.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:50.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:51.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:52.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:52.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:52.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:53:52.055 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:53:52.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:52.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:52.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:53:52.153+0000 7f3cda80e4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:53:52.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:53.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:53.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:53.326 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:53:53.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:53:53.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:53.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:53.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:54.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:54.600 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:53:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:54.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:53:54.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:54.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:54.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:55.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:55.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:55.878 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:53:55.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:53:55.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:55.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:56.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:57.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:57.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:57.188 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:53:57.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:53:57.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:57.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:57.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:58.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:58.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:58.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:53:58.458 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:53:58.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:58.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:53:58.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:53:59.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:53:59.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:53:59.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:53:59.733 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:53:59.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:53:59.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:00.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:01.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:01.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:01.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:54:01.012 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:54:01.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:01.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:01.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:02.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:02.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:02.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:54:02.288 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:54:02.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:02.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:02.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:03.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:03.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:03.567 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:54:03.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:54:03.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:03.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:04.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:05.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:05.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:05.080 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:54:05.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:54:05.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:05.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:05.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:06.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:06.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:06.547 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:54:06.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:54:06.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:06.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:06.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:07.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:07.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:07.954 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:54:07.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:54:07.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:07.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:08.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:09.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:09.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:09.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:54:09.400 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:54:09.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:09.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:10.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:11.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:11.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:11.144 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:54:11.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:54:11.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:11.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:11.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:12.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:12.915 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:54:12.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:54:12.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:12.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:13.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:14.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:14.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:14.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:54:14.685 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:54:14.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:14.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:54:14.992 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1716380865,v1:127.0.0.1:6803/1716380865] [v2:127.0.0.1:6804/1716380865,v1:127.0.0.1:6805/1716380865] exists,up 12e33266-2d9a-4dc8-b7b4-c1b67ae2f8d6 2024-02-05T05:54:14.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:54:14.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:54:14.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:54:14.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:54:14.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ac98ee6-a8cf-4403-9728-d32b031a75bf 2024-02-05T05:54:14.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3ac98ee6-a8cf-4403-9728-d32b031a75bf' 2024-02-05T05:54:15.000 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 3ac98ee6-a8cf-4403-9728-d32b031a75bf 2024-02-05T05:54:15.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:54:15.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHeMBlzGjmARAAn8xCI0NLjpGJl6Zf/gS/pQ== 2024-02-05T05:54:15.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHeMBlzGjmARAAn8xCI0NLjpGJl6Zf/gS/pQ=="}' 2024-02-05T05:54:15.014 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ac98ee6-a8cf-4403-9728-d32b031a75bf -i td/osd-scrub-repair/1/new.json 2024-02-05T05:54:15.434 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:54:15.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:54:15.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAHeMBlzGjmARAAn8xCI0NLjpGJl6Zf/gS/pQ== --osd-uuid 3ac98ee6-a8cf-4403-9728-d32b031a75bf 2024-02-05T05:54:15.476 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:15.493+0000 7f26a66cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:15.529 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:15.546+0000 7f26a66cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:15.529 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:15.547+0000 7f26a66cf4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:15.530 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:15.548+0000 7f26a66cf4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:54:15.530 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:15.548+0000 7f26a66cf4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:54:15.530 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:15.548+0000 7f26a66cf4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:54:15.602 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:15.619+0000 7f26a66cf4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:54:21.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:54:21.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:54:21.057 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:54:21.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:54:21.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:54:21.403 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:54:21.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:54:21.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:54:21.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:54:21.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:54:21.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:54:21.430 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:21.448+0000 7fe3c1c2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:21.432 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:21.450+0000 7fe3c1c2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:21.433 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:21.451+0000 7fe3c1c2a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:54:21.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:54:21.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:54:21.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:54:21.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:54:21.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:54:21.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:21.716 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:54:21.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:54:21.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:21.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:21.956 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:21.974+0000 7fe3c1c2a4c0 -1 Falling back to public interface 2024-02-05T05:54:22.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:23.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:23.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:23.028 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:54:23.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:54:23.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:23.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:23.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:24.300 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:54:24.318+0000 7fe3c1c2a4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:54:24.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:24.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:24.307 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:54:24.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:54:24.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:24.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:24.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:25.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:25.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:25.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:54:25.602 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:54:25.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:25.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:26.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:27.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:27.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:27.065 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:54:27.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:54:27.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:27.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:27.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:28.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:28.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:28.342 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:54:28.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:54:28.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:28.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:28.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:29.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:29.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:29.621 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:54:29.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:54:29.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:29.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:29.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:30.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:30.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:30.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:54:30.901 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:54:30.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:30.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:31.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:32.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:32.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:32.284 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:54:32.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:54:32.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:32.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:32.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:33.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:33.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:33.563 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:54:33.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:54:33.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:33.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:33.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:34.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:34.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:34.896 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:54:34.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:54:34.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:34.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:35.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:36.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:36.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:36.318 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:54:36.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:54:36.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:36.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:36.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:37.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:37.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:37.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:54:37.595 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:54:37.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:37.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:37.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:38.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:38.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:38.963 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:54:38.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:54:38.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:38.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:39.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:40.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:40.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:40.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:54:40.387 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:54:40.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:40.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:40.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:41.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:41.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:41.949 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:54:41.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:54:41.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:41.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:42.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:43.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:43.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:43.249 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:54:43.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:54:43.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:43.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:44.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:44.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:44.977 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:54:44.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:54:44.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:44.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:45.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:46.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:46.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:46.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:54:46.404 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:54:46.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:46.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:47.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:54:48.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:54:48.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:54:48.062 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:54:48.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:54:48.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:54:48.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:54:48.337 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4162426452,v1:127.0.0.1:6811/4162426452] [v2:127.0.0.1:6812/4162426452,v1:127.0.0.1:6813/4162426452] exists,up 3ac98ee6-a8cf-4403-9728-d32b031a75bf 2024-02-05T05:54:48.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:54:48.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:54:48.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:54:48.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-02-05T05:54:48.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-02-05T05:54:48.808 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'testpool' created 2024-02-05T05:54:48.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:54:49.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-02-05T05:54:50.230 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 size to 2 2024-02-05T05:54:50.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-02-05T05:54:50.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:54:50.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:54:50.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:54:50.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:54:50.243 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:54:50.243 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:54:50.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:54:50.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:54:50.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:54:50.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:54:50.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:54:50.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:54:50.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:54:50.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:54:50.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:54:50.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:54:50.808 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:54:50.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:54:50.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:50.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:54:50.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-02-05T05:54:50.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-02-05T05:54:50.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-02-05T05:54:50.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:54:50.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:54:51.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T05:54:51.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T05:54:51.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672964' 2024-02-05T05:54:51.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:51.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-02-05T05:54:51.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:51.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:54:51.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-02-05T05:54:51.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:51.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-02-05T05:54:51.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-02-05T05:54:51.039 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836493 2024-02-05T05:54:51.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:51.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-02-05T05:54:51.344 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:54:52.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:54:52.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:52.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-02-05T05:54:52.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:54:53.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:54:53.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:54:53.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-02-05T05:54:53.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:54:53.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T05:54:53.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:54:53.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:54:53.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T05:54:53.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:54:53.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T05:54:53.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T05:54:53.900 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672964 2024-02-05T05:54:53.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:54:54.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-02-05T05:54:54.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:54:54.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:54.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:54:54.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:54:54.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:54:54.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:54:54.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:54:54.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:54:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:54:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:54:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:54:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-02-05T05:54:55.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-02-05T05:54:55.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-02-05T05:54:55.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:55.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:55.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:55.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:55.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:55.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:55.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:55.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:56.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:56.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:56.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:56.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:56.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:56.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:56.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:56.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:56.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:56.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:57.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:57.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:58.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:58.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:58.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:58.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:58.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:58.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:58.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:58.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:58.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:58.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:58.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:58.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:58.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-02-05T05:54:58.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T05:54:59.004 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:54:59.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:54:59.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-02-05T05:54:59.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:54:59.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-02-05T05:54:59.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-02-05T05:54:59.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T05:54:59.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T05:54:59.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T05:54:59.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:54:59.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-02-05T05:54:59.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-02-05T05:54:59.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:54:59.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T05:54:59.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T05:54:59.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:55:00.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-02-05T05:55:00.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:55:00.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:55:00.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:55:00.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:55:00.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:55:00.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:55:00.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:55:00.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:55:00.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:55:00.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:55:00.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:55:00.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:55:00.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:55:00.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:55:00.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-02-05T05:55:00.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:00.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-02-05T05:55:00.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:00.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-02-05T05:55:00.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:00.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:00.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:00.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:00.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:00.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-02-05T05:55:01.888 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T05:55:02.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:02.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-02-05T05:55:02.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:02.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:55:02.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:02.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:02.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:02.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:02.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:02.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T05:55:03.571 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:104778fc:::obj2:head# 2024-02-05T05:55:04.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:04.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-02-05T05:55:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-02-05T05:55:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:04.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:04.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-02-05T05:55:04.845 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:8dd16f86:::obj3:head# 2024-02-05T05:55:05.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:05.363 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-02-05T05:55:05.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:05.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-02-05T05:55:05.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:05.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:05.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:05.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:05.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:05.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-02-05T05:55:06.131 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:0ee9ae15:::obj4:head# 2024-02-05T05:55:06.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:06.649 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:06.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-02-05T05:55:07.400 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:c52c9666:::obj5:head# 2024-02-05T05:55:07.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:07.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:07.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-02-05T05:55:08.594 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:233a42c1:::obj6:head# 2024-02-05T05:55:09.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:09.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-02-05T05:55:09.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:09.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-02-05T05:55:09.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:09.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:09.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:09.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:09.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:09.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-02-05T05:55:09.763 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:9a09113a:::obj7:head# 2024-02-05T05:55:10.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:10.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-02-05T05:55:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-02-05T05:55:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:10.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:10.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:10.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-02-05T05:55:10.914 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:a557efb1:::obj8:head# 2024-02-05T05:55:11.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:11.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:11.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-02-05T05:55:12.067 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:fe351e27:::obj9:head# 2024-02-05T05:55:12.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:12.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:12.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-02-05T05:55:13.302 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:d1337354:::obj10:head# 2024-02-05T05:55:13.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:13.817 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:13.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-02-05T05:55:14.463 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:eeae2a94:::obj11:head# 2024-02-05T05:55:15.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:15.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-02-05T05:55:15.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:15.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-02-05T05:55:15.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:15.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:15.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:15.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:15.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:15.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-02-05T05:55:15.882 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:af5f95cb:::obj12:head# 2024-02-05T05:55:16.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:16.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-02-05T05:55:16.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:16.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-02-05T05:55:16.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:16.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:16.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:16.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:16.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:16.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-02-05T05:55:17.369 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:a61ad63e:::obj13:head# 2024-02-05T05:55:17.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:17.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:17.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-02-05T05:55:18.478 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:e0a44829:::obj14:head# 2024-02-05T05:55:18.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:18.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:18.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-02-05T05:55:19.622 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:ab946124:::obj15:head# 2024-02-05T05:55:20.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:20.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:20.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-02-05T05:55:20.748 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:461f8b5e:::obj16:head# 2024-02-05T05:55:21.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:21.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:21.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-02-05T05:55:21.883 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:24fc6e92:::obj17:head# 2024-02-05T05:55:22.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:22.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:22.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-02-05T05:55:23.060 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:b9836a99:::obj18:head# 2024-02-05T05:55:23.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:23.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:23.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-02-05T05:55:24.212 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:00b31b6c:::obj19:head# 2024-02-05T05:55:24.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-02-05T05:55:24.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:24.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-02-05T05:55:25.447 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:24b5611b:::obj20:head# 2024-02-05T05:55:25.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:55:25.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:25.968 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:55:25.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:55:25.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:55:25.970 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:55:25.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:55:25.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:55:25.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:55:25.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:55:25.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:55:25.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:55:25.997 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:26.015+0000 7f72c8c104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:25.999 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:26.018+0000 7f72c8c104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:26.000 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:26.019+0000 7f72c8c104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:26.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:55:26.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:55:26.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:55:26.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:55:26.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:55:26.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:26.245 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:55:26.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:55:26.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:26.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:26.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:26.522 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:26.541+0000 7f72c8c104c0 -1 Falling back to public interface 2024-02-05T05:55:27.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:27.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:27.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:55:27.511 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:55:27.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:27.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:27.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:28.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:28.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:28.784 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:55:28.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:55:28.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:28.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:28.788 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:28.807+0000 7f72c8c104c0 -1 osd.1 16 log_to_monitors true 2024-02-05T05:55:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:30.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:30.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:30.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:55:30.065 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:55:30.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:30.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:55:30.335 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 19 up_thru 19 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6802/4229469928,v1:127.0.0.1:6803/4229469928] [v2:127.0.0.1:6804/4229469928,v1:127.0.0.1:6805/4229469928] exists,up 3ac98ee6-a8cf-4403-9728-d32b031a75bf 2024-02-05T05:55:30.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:55:30.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:55:30.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:55:30.341 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:55:30.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:55:30.342 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:55:30.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:55:30.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:55:30.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:55:30.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:55:30.365 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:30.383+0000 7f65c64104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:30.367 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:30.385+0000 7f65c64104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:30.367 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:30.386+0000 7f65c64104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:55:30.606 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:55:30.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:30.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:30.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:30.887 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:30.906+0000 7f65c64104c0 -1 Falling back to public interface 2024-02-05T05:55:31.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:31.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:31.871 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:55:31.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:55:31.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:31.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:32.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:33.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:33.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:33.148 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:55:33.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:55:33.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:33.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:33.328 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:33.347+0000 7f65c64104c0 -1 osd.0 15 log_to_monitors true 2024-02-05T05:55:33.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:55:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:55:34.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:55:34.434 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:55:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:55:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:55:34.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:55:34.706 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 16 last_clean_interval [5,15) [v2:127.0.0.1:6810/3910369443,v1:127.0.0.1:6811/3910369443] [v2:127.0.0.1:6812/3910369443,v1:127.0.0.1:6813/3910369443] exists,up 12e33266-2d9a-4dc8-b7b4-c1b67ae2f8d6 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:55:34.708 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:55:34.709 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:55:34.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:55:34.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:55:34.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:55:34.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:55:34.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:55:34.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:55:34.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:55:34.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:55:34.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:55:35.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:55:35.065 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:55:35.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:55:35.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:35.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:55:35.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=94489280514 2024-02-05T05:55:35.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 94489280514 2024-02-05T05:55:35.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-94489280514' 2024-02-05T05:55:35.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:35.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:55:35.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378627 2024-02-05T05:55:35.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378627 2024-02-05T05:55:35.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-94489280514 1-81604378627' 2024-02-05T05:55:35.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:35.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-94489280514 2024-02-05T05:55:35.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:35.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:55:35.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-94489280514 2024-02-05T05:55:35.301 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:35.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=94489280514 2024-02-05T05:55:35.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2024-02-05T05:55:35.303 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 94489280514 2024-02-05T05:55:35.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:35.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 94489280514 2024-02-05T05:55:35.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:55:36.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:55:36.572 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:36.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 94489280514 2024-02-05T05:55:36.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:55:37.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:55:37.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:38.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 94489280514 -lt 94489280514 2024-02-05T05:55:38.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:38.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-81604378627 2024-02-05T05:55:38.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:38.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:55:38.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-81604378627 2024-02-05T05:55:38.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:38.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378627 2024-02-05T05:55:38.115 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 81604378627 2024-02-05T05:55:38.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 81604378627' 2024-02-05T05:55:38.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:55:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378627 -lt 81604378627 2024-02-05T05:55:38.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:55:38.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:55:38.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:55:38.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:55:38.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:55:38.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:55:38.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:55:38.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:55:38.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:55:38.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:55:38.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:55:38.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:55:38.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:55:38.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:55:38.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:55:39.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:39.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:39.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:55:39.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:39.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T05:55:39.827 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0 on osd.1 to repair 2024-02-05T05:55:39.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:39.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:55:39.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:39.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:55:39.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:55:39.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:39.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:55:39.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:55:39.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:39.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:39.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:55:40.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:48.751067+0000 '>' 2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:40.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:41.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:41.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:41.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:55:41.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:55:41.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:41.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:41.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:55:41.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:48.751067+0000 '>' 2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:41.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:42.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:42.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:42.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:55:42.442 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:55:42.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:42.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:42.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:55:42.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:48.751067+0000 '>' 2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:42.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:43.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:55:43.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:55:43.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:43.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:43.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:55:43.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:54:48.751067+0000 '>' 2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:43.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:55:44.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:55:44.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:55:44.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:55:44.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:55:44.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:55:44.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:55:44.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:55:45.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:55:40.742055+0000 '>' 2024-02-05T05:54:48.751067+0000 2024-02-05T05:55:45.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:55:45.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:55:45.248 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:55:45.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:55:45.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:55:45.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:55:45.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:55:45.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:55:45.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:55:45.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:55:45.604 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:55:45.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:55:45.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:45.604 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:55:45.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=94489280517 2024-02-05T05:55:45.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 94489280517 2024-02-05T05:55:45.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-94489280517' 2024-02-05T05:55:45.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:45.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:55:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378630 2024-02-05T05:55:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378630 2024-02-05T05:55:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-94489280517 1-81604378630' 2024-02-05T05:55:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:45.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-94489280517 2024-02-05T05:55:45.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:55:45.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-94489280517 2024-02-05T05:55:45.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:45.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=94489280517 2024-02-05T05:55:45.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 94489280517' 2024-02-05T05:55:45.815 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 94489280517 2024-02-05T05:55:45.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:46.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 94489280516 -lt 94489280517 2024-02-05T05:55:46.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:55:47.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:55:47.094 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:47.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 94489280517 -lt 94489280517 2024-02-05T05:55:47.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:47.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-81604378630 2024-02-05T05:55:47.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:47.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:55:47.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-81604378630 2024-02-05T05:55:47.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:47.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378630 2024-02-05T05:55:47.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 81604378630' 2024-02-05T05:55:47.368 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 81604378630 2024-02-05T05:55:47.368 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:55:47.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378630 -lt 81604378630 2024-02-05T05:55:47.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:55:47.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:55:47.638 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:55:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:55:47.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:55:47.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:55:47.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:55:47.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:55:47.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:55:47.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:55:47.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:55:48.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:55:48.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:55:48.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:55:48.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:55:48.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:55:48.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:55:48.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:55:48.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-02-05T05:55:48.823 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:55:48.823 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-02-05T05:55:41.773174+0000 15'30 23:118 [1,0] 1 [1,0] 1 15'30 2024-02-05T05:55:40.742055+0000 15'30 2024-02-05T05:55:40.742055+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:55:40.742055+0000 20 0 2024-02-05T05:55:48.823 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T05:55:48.823 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:55:48.823 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T05:55:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-02-05T05:55:48.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:55:48.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:55:49.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:55:49.098 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T05:55:49.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:55:49.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:49.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:55:49.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=94489280519 2024-02-05T05:55:49.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 94489280519 2024-02-05T05:55:49.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-94489280519' 2024-02-05T05:55:49.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:55:49.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:55:49.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378632 2024-02-05T05:55:49.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378632 2024-02-05T05:55:49.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-94489280519 1-81604378632' 2024-02-05T05:55:49.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:49.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-94489280519 2024-02-05T05:55:49.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:49.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:55:49.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-94489280519 2024-02-05T05:55:49.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:49.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=94489280519 2024-02-05T05:55:49.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 94489280519' 2024-02-05T05:55:49.322 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 94489280519 2024-02-05T05:55:49.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:49.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 94489280518 -lt 94489280519 2024-02-05T05:55:49.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:55:50.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:55:50.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:50.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 94489280518 -lt 94489280519 2024-02-05T05:55:50.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:55:51.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T05:55:51.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:55:52.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 94489280519 -lt 94489280519 2024-02-05T05:55:52.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:55:52.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-81604378632 2024-02-05T05:55:52.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:55:52.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:55:52.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-81604378632 2024-02-05T05:55:52.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:55:52.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378632 2024-02-05T05:55:52.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 81604378632' 2024-02-05T05:55:52.148 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 81604378632 2024-02-05T05:55:52.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:55:52.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378632 -lt 81604378632 2024-02-05T05:55:52.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-02-05T05:55:52.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-02-05T05:55:52.520 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:55:52.520 INFO:tasks.workunit.client.0.smithi188.stdout: "num_bytes": 210, 2024-02-05T05:55:52.520 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects": 30, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_object_clones": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_object_copies": 60, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_missing_on_primary": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_missing": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_degraded": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_misplaced": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_unfound": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_dirty": 30, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_whiteouts": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_read": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_read_kb": 0, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_write": 30, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_write_kb": 30, 2024-02-05T05:55:52.521 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrub_errors": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shallow_scrub_errors": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_deep_scrub_errors": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_recovered": 20, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_bytes_recovered": 140, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_keys_recovered": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_omap": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_hit_set_archive": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_bytes_hit_set_archive": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush_kb": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict_kb": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_promote": 0, 2024-02-05T05:55:52.522 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush_mode_high": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush_mode_low": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict_mode_some": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict_mode_full": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_pinned": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_legacy_snapsets": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_large_omap_objects": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_manifest": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_omap_bytes": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_omap_keys": 0, 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_repaired": 20 2024-02-05T05:55:52.523 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:55:52.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-02-05T05:55:52.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-02-05T05:55:52.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-02-05T05:55:52.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-02-05T05:55:52.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T05:55:52.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-02-05T05:55:52.876 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:55:52.887 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:55:52.887 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T05:55:52.887 INFO:tasks.workunit.client.0.smithi188.stdout: "up_from": 19, 2024-02-05T05:55:52.887 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 81604378632, 2024-02-05T05:55:52.887 INFO:tasks.workunit.client.0.smithi188.stdout: "num_pgs": 1, 2024-02-05T05:55:52.887 INFO:tasks.workunit.client.0.smithi188.stdout: "num_osds": 1, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_osds": 1, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "kb": 104857600, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used": 28544, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_data": 312, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_omap": 4, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_meta": 28219, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_avail": 104829056, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "statfs": { 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "total": 107374182400, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "available": 107344953344, 2024-02-05T05:55:52.888 INFO:tasks.workunit.client.0.smithi188.stdout: "internally_reserved": 0, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "allocated": 319488, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "data_stored": 50951, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed": 0, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_allocated": 0, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_original": 0, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_allocated": 4397, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "internal_metadata": 28896979 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "hb_peers": [ 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: 0 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "snap_trim_queue_len": 0, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "num_snap_trimming": 0, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shards_repaired": 10, 2024-02-05T05:55:52.889 INFO:tasks.workunit.client.0.smithi188.stdout: "op_queue_age_hist": { 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: "histogram": [], 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: "upper_bound": 1 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: "perf_stat": { 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ms": 0, 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ms": 0, 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ns": 0, 2024-02-05T05:55:52.890 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ns": 0 2024-02-05T05:55:52.891 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:52.891 INFO:tasks.workunit.client.0.smithi188.stdout: "alerts": [], 2024-02-05T05:55:52.891 INFO:tasks.workunit.client.0.smithi188.stdout: "network_ping_times": [ 2024-02-05T05:55:52.891 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:55:52.891 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T05:55:52.891 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:55:52.891 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:55:52.892 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.222 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:55:52.893 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:55:52.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T05:55:52.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-02-05T05:55:53.141 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:55:53.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-02-05T05:55:53.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-02-05T05:55:53.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-02-05T05:55:53.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T05:55:53.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-02-05T05:55:53.405 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:55:53.416 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:55:53.416 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "up_from": 22, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 94489280519, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "num_pgs": 1, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "num_osds": 1, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_osds": 1, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "kb": 104857600, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used": 28544, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_data": 312, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_omap": 4, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_meta": 28219, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_avail": 104829056, 2024-02-05T05:55:53.417 INFO:tasks.workunit.client.0.smithi188.stdout: "statfs": { 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "total": 107374182400, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "available": 107344953344, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "internally_reserved": 0, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "allocated": 319488, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "data_stored": 50951, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed": 0, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_allocated": 0, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_original": 0, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_allocated": 4401, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "internal_metadata": 28896975 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: "hb_peers": [ 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T05:55:53.418 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "snap_trim_queue_len": 0, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "num_snap_trimming": 0, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shards_repaired": 10, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "op_queue_age_hist": { 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "histogram": [], 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "upper_bound": 1 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "perf_stat": { 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ms": 0, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ms": 0, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ns": 0, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ns": 0 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.419 INFO:tasks.workunit.client.0.smithi188.stdout: "alerts": [], 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "network_ping_times": [ 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:55:53.420 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.16600000000000001 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:55:53.421 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:55:53.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T05:55:53.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-02-05T05:55:53.668 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:55:53.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-02-05T05:55:53.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-02-05T05:55:53.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-02-05T05:55:53.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T05:55:53.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-02-05T05:55:53.932 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "up_from": 0, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "num_pgs": 2, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "num_osds": 2, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_osds": 2, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_omap_osds": 2, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "kb": 209715200, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used": 57088, 2024-02-05T05:55:53.943 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_data": 624, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_omap": 8, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_meta": 56439, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_avail": 209658112, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "statfs": { 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "total": 214748364800, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "available": 214689906688, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "internally_reserved": 0, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "allocated": 638976, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "data_stored": 101902, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed": 0, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_allocated": 0, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_original": 0, 2024-02-05T05:55:53.944 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_allocated": 8798, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "internal_metadata": 57793954 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "hb_peers": [], 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "snap_trim_queue_len": 0, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "num_snap_trimming": 0, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shards_repaired": 20, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "op_queue_age_hist": { 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "histogram": [], 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "upper_bound": 1 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "perf_stat": { 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ms": 0, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ms": 0, 2024-02-05T05:55:53.945 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ns": 0, 2024-02-05T05:55:53.946 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ns": 0 2024-02-05T05:55:53.946 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:55:53.946 INFO:tasks.workunit.client.0.smithi188.stdout: "alerts": [], 2024-02-05T05:55:53.946 INFO:tasks.workunit.client.0.smithi188.stdout: "network_ping_times": [] 2024-02-05T05:55:53.946 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:55:53.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-02-05T05:55:53.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-02-05T05:55:54.193 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:55:54.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-02-05T05:55:54.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-02-05T05:55:54.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:55:54.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:55:54.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:55:54.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:55:54.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:55:54.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:55:54.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:55:54.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:55:54.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:55:54.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:55:54.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:55:54.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:55:54.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:55:54.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:55:54.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:55:54.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:55:54.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:54.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:55:54.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:55:54.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:54.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:55:54.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:55:54.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:55:54.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:55:54.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:54.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:54.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:55:54.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:55:54.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:55:54.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:55:54.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:55:54.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:55:54.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:55:54.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:55:54.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:55:54.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:55:54.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:55:54.339 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:55:54.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:55:54.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:55:54.340 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:55:54.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:54.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:55:54.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:55:54.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:55:54.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:55:54.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:55:54.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:55:54.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:55:54.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:54.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:54.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:55:54.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:55:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:55:54.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:55:54.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:55:54.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:54.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:54.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-02-05T05:55:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:55:54.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T05:55:54.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:55:54.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:54.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:54.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:54.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:54.768 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:54.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:54.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T05:55:54.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:55:54.953 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:55:54.954 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:55:54.954 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:54.954 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:54.954 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:55:54.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:55:54.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:55:55.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:55:55.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:55:55.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:55:55.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:55:55.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:55:55.034 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:55:55.034 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:55:55.034 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:55:55.035 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:55:55.035 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:55.035 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:55.035 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:55:55.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:55:55.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:55:55.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-02-05T05:55:55.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:55:55.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:55:55.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:55:55.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:55:55.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:55:55.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:55:55.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:55:55.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:55.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:55.921 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:55.921 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:55.922 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:55.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:55.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:55:55.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:55:55.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:55:55.948 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-02-05T05:55:55.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-02-05T05:55:55.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:55:55.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:55:55.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:55.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=792b9be4-8ee3-409e-b2b8-e73459e8119e 2024-02-05T05:55:55.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 792b9be4-8ee3-409e-b2b8-e73459e8119e' 2024-02-05T05:55:55.958 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 792b9be4-8ee3-409e-b2b8-e73459e8119e 2024-02-05T05:55:55.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:55:55.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBreMBlnjoaOxAAiqgsi3eP5IyYhycRLAuB3g== 2024-02-05T05:55:55.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBreMBlnjoaOxAAiqgsi3eP5IyYhycRLAuB3g=="}' 2024-02-05T05:55:55.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 792b9be4-8ee3-409e-b2b8-e73459e8119e -i td/osd-scrub-repair/0/new.json 2024-02-05T05:55:56.218 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:55:56.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:55:56.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBreMBlnjoaOxAAiqgsi3eP5IyYhycRLAuB3g== --osd-uuid 792b9be4-8ee3-409e-b2b8-e73459e8119e 2024-02-05T05:55:56.259 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:56.277+0000 7f64732b64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:56.269 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:56.288+0000 7f64732b64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:56.270 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:56.289+0000 7f64732b64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:55:56.271 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:56.290+0000 7f64732b64c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:55:56.271 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:56.290+0000 7f64732b64c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:55:56.271 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:56.290+0000 7f64732b64c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:55:56.361 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:55:56.380+0000 7f64732b64c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:56:01.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:56:01.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:56:01.821 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:56:01.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:56:01.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:56:02.171 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:56:02.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:56:02.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:56:02.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:56:02.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:56:02.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:56:02.198 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:02.217+0000 7fd7565a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:02.200 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:02.219+0000 7fd7565a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:02.201 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:02.220+0000 7fd7565a94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:02.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:56:02.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:56:02.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:56:02.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:56:02.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:56:02.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:02.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:56:02.438 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:56:02.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:02.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:02.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:02.728 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:02.748+0000 7fd7565a94c0 -1 Falling back to public interface 2024-02-05T05:56:03.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:03.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:03.707 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:56:03.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:56:03.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:03.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:03.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:04.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:04.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:04.986 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:56:04.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:56:04.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:04.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:05.152 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:05.171+0000 7fd7565a94c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:56:05.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:06.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:06.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:06.265 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:56:06.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:56:06.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:06.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:06.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:07.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:07.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:07.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:56:07.544 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:56:07.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:07.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:07.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:08.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:08.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:08.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:56:08.940 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:56:08.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:08.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:09.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:10.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:10.215 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:56:10.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:10.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:56:10.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:10.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:10.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:11.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:11.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:11.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:56:11.494 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:56:11.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:11.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:11.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:12.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:12.771 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:56:12.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:56:12.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:12.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:13.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:14.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:14.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:14.148 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:56:14.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:56:14.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:14.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:14.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:15.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:15.426 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:56:15.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:15.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:56:15.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:15.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:15.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:16.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:16.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:16.731 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:56:16.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:56:16.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:16.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:17.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:18.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:18.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:18.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:56:18.006 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:56:18.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:18.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:18.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:19.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:19.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:19.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:56:19.309 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:56:19.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:19.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:19.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:20.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:20.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:20.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:56:20.829 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:56:20.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:20.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:21.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:22.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:22.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:22.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:56:22.266 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:56:22.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:22.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:23.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:24.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:24.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:24.027 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:56:24.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:56:24.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:24.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:24.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:25.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:25.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:25.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:56:25.755 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:56:25.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:25.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:26.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:27.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:27.504 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:56:27.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:27.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:56:27.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:27.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:56:27.791 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/472833206,v1:127.0.0.1:6803/472833206] [v2:127.0.0.1:6804/472833206,v1:127.0.0.1:6805/472833206] exists,up 792b9be4-8ee3-409e-b2b8-e73459e8119e 2024-02-05T05:56:27.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:56:27.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:56:27.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:56:27.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:56:27.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:56:27.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:56:27.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:56:27.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c25ce3a7-548b-4db6-8195-bd882f822f2f 2024-02-05T05:56:27.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c25ce3a7-548b-4db6-8195-bd882f822f2f' 2024-02-05T05:56:27.800 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 c25ce3a7-548b-4db6-8195-bd882f822f2f 2024-02-05T05:56:27.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:56:27.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLeMBlJWqlMRAADYV3EwbhvxHRI5ywf4QM3w== 2024-02-05T05:56:27.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLeMBlJWqlMRAADYV3EwbhvxHRI5ywf4QM3w=="}' 2024-02-05T05:56:27.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c25ce3a7-548b-4db6-8195-bd882f822f2f -i td/osd-scrub-repair/1/new.json 2024-02-05T05:56:28.207 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:56:28.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T05:56:28.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCLeMBlJWqlMRAADYV3EwbhvxHRI5ywf4QM3w== --osd-uuid c25ce3a7-548b-4db6-8195-bd882f822f2f 2024-02-05T05:56:28.248 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:28.268+0000 7f5332e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:28.292 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:28.311+0000 7f5332e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:28.292 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:28.312+0000 7f5332e164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:28.293 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:28.312+0000 7f5332e164c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:56:28.293 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:28.312+0000 7f5332e164c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:56:28.293 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:28.312+0000 7f5332e164c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T05:56:28.350 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:28.369+0000 7f5332e164c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T05:56:34.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T05:56:34.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:56:34.351 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T05:56:34.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T05:56:34.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:56:34.696 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:56:34.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T05:56:34.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:56:34.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:56:34.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:56:34.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:56:34.723 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:34.743+0000 7f9f1d9fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:34.725 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:34.745+0000 7f9f1d9fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:34.726 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:34.746+0000 7f9f1d9fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:56:34.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T05:56:34.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:56:34.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:56:34.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:56:34.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:56:34.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:34.966 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:56:34.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:56:34.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:34.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:35.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:35.253 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:35.273+0000 7f9f1d9fe4c0 -1 Falling back to public interface 2024-02-05T05:56:36.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:36.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:36.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:56:36.235 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:56:36.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:36.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:36.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:37.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:37.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:37.511 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:56:37.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:56:37.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:37.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:37.575 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:56:37.595+0000 7f9f1d9fe4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T05:56:37.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:38.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:38.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:38.785 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:56:38.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:56:38.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:38.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:39.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:40.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:40.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:40.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:56:40.068 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:56:40.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:40.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:40.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:41.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:41.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:41.347 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:56:41.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:56:41.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:41.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:41.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:42.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:42.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:42.629 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:56:42.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:56:42.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:42.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:42.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:43.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:43.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:43.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:56:43.906 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:56:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:44.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:45.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:45.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:45.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:56:45.298 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:56:45.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:45.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:45.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:46.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:46.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:46.577 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:56:46.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:56:46.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:46.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:46.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:47.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:47.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:47.856 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:56:47.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:56:47.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:47.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:48.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:49.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:49.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:49.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:56:49.256 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:56:49.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:49.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:49.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:50.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:50.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:50.806 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:56:50.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:56:50.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:50.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:51.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:52.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:52.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:52.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:56:52.428 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:56:52.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:52.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:52.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:53.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:53.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:53.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:56:53.846 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:56:53.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:53.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:54.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:55.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:55.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:55.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:56:55.256 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:56:55.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:55.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:55.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:56.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:56.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:56.973 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:56:56.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:56:56.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:56.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:57.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:56:58.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:56:58.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:56:58.703 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:56:58.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:56:58.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:56:58.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:56:59.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:00.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:00.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:00.270 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:57:00.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:57:00.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:00.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:57:00.581 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/178196156,v1:127.0.0.1:6811/178196156] [v2:127.0.0.1:6812/178196156,v1:127.0.0.1:6813/178196156] exists,up c25ce3a7-548b-4db6-8195-bd882f822f2f 2024-02-05T05:57:00.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:57:00.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:57:00.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T05:57:00.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:57:00.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a731b258-8de2-424f-97f7-935513afb370 2024-02-05T05:57:00.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a731b258-8de2-424f-97f7-935513afb370' 2024-02-05T05:57:00.589 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 a731b258-8de2-424f-97f7-935513afb370 2024-02-05T05:57:00.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:57:00.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCseMBlf6AgJRAAe3EmmEB8QskD+4ZmtpOqVQ== 2024-02-05T05:57:00.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCseMBlf6AgJRAAe3EmmEB8QskD+4ZmtpOqVQ=="}' 2024-02-05T05:57:00.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a731b258-8de2-424f-97f7-935513afb370 -i td/osd-scrub-repair/2/new.json 2024-02-05T05:57:01.034 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:57:01.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T05:57:01.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCseMBlf6AgJRAAe3EmmEB8QskD+4ZmtpOqVQ== --osd-uuid a731b258-8de2-424f-97f7-935513afb370 2024-02-05T05:57:01.076 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:01.096+0000 7f809a08c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:01.118 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:01.138+0000 7f809a08c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:01.119 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:01.139+0000 7f809a08c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:01.120 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:01.140+0000 7f809a08c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:57:01.120 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:01.140+0000 7f809a08c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:57:01.120 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:01.140+0000 7f809a08c4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T05:57:01.193 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:01.213+0000 7f809a08c4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T05:57:07.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T05:57:07.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:57:07.334 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T05:57:07.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T05:57:07.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:57:07.677 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T05:57:07.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T05:57:07.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:57:07.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:57:07.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:57:07.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:57:07.705 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:07.725+0000 7f4dc260d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:07.707 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:07.727+0000 7f4dc260d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:07.708 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:07.728+0000 7f4dc260d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:57:07.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T05:57:07.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:57:07.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T05:57:07.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:57:07.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:57:07.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:07.947 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:57:07.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:57:07.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:07.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:08.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:08.240 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:08.260+0000 7f4dc260d4c0 -1 Falling back to public interface 2024-02-05T05:57:09.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:09.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:09.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:57:09.216 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:57:09.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:09.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:09.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:10.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:10.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:10.491 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:57:10.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:57:10.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:10.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:10.567 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:57:10.587+0000 7f4dc260d4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T05:57:10.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:11.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:11.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:11.794 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:57:11.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:57:11.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:11.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:12.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:13.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:13.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:13.114 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:57:13.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:57:13.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:13.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:13.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:14.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:14.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:14.420 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:57:14.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:57:14.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:14.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:14.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:15.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:15.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:15.698 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:57:15.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:57:15.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:15.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:15.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:16.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:16.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:16.979 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:57:16.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:57:16.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:16.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:17.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:18.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:18.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:18.259 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:57:18.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:57:18.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:18.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:18.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:19.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:19.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:19.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:57:19.614 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:57:19.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:19.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:19.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:20.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:20.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:20.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:57:20.891 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:57:20.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:20.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:21.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:22.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:22.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:22.172 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:57:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:57:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:22.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:22.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:23.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:23.492 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:57:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:57:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:23.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:24.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:25.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:25.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:25.008 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:57:25.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:57:25.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:25.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:25.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:26.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:26.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:26.611 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:57:26.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:57:26.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:26.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:26.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:27.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:27.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:27.961 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:57:27.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:57:27.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:27.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:28.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:29.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:29.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:29.765 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:57:29.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:57:29.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:29.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:30.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:31.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:31.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:31.280 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:57:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:57:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:31.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:32.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:33.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:33.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:33.063 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:57:33.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:57:33.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:33.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:33.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:57:34.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:57:34.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:57:34.455 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:57:34.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:57:34.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:57:34.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T05:57:34.758 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/193635599,v1:127.0.0.1:6819/193635599] [v2:127.0.0.1:6820/193635599,v1:127.0.0.1:6821/193635599] exists,up a731b258-8de2-424f-97f7-935513afb370 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T05:57:34.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-02-05T05:57:35.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-02-05T05:57:35.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-02-05T05:57:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'testpool' created 2024-02-05T05:57:35.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:57:36.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T05:57:36.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T05:57:36.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:57:36.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:57:36.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:57:36.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:57:36.926 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:57:36.926 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:57:36.926 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:57:36.926 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:57:36.926 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:57:37.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:57:37.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:57:37.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:57:37.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:57:37.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:57:37.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:57:37.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:57:37.348 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:57:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:57:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:57:37.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:37.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:57:37.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-02-05T05:57:37.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-02-05T05:57:37.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-02-05T05:57:37.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:37.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:57:37.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-02-05T05:57:37.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-02-05T05:57:37.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973' 2024-02-05T05:57:37.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:57:37.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:57:37.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-02-05T05:57:37.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-02-05T05:57:37.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672973 2-64424509447' 2024-02-05T05:57:37.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:37.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-02-05T05:57:37.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:37.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:57:37.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-02-05T05:57:37.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:37.659 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836499 2024-02-05T05:57:37.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-02-05T05:57:37.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-02-05T05:57:37.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:37.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-02-05T05:57:37.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:57:38.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:57:38.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:57:39.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-02-05T05:57:39.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:39.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-02-05T05:57:39.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:39.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:57:39.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-02-05T05:57:39.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:39.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-02-05T05:57:39.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-02-05T05:57:39.209 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672973 2024-02-05T05:57:39.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:57:39.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-02-05T05:57:39.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:57:39.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-02-05T05:57:39.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:57:39.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:57:39.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-02-05T05:57:39.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:57:39.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-02-05T05:57:39.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-02-05T05:57:39.478 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509447 2024-02-05T05:57:39.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:57:39.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-02-05T05:57:39.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:57:39.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:39.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:40.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:57:40.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:40.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:40.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:40.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:40.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:40.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:40.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:40.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:57:40.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:40.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:40.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:40.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T05:57:40.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T05:57:40.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T05:57:40.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T05:57:40.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:57:40.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T05:57:40.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:57:40.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:40.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:40.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:40.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:40.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:40.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:40.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:41.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T05:57:41.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:41.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:41.049 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-02-05T05:57:41.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-02-05T05:57:41.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-02-05T05:57:41.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-02-05T05:57:41.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-02-05T05:57:41.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-02-05T05:57:41.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T05:57:41.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-02-05T05:57:41.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T05:57:41.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:57:41.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:57:41.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:57:41.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:57:41.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:57:41.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:57:41.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:57:42.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:57:42.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:57:42.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:57:42.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:57:42.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:57:42.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:57:42.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:57:42.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T05:57:42.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-02-05T05:57:42.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-02-05T05:57:42.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-02-05T05:57:42.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:42.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:42.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:42.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:42.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:42.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:43.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:43.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:43.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:43.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:43.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:43.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:43.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:43.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:43.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:43.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:44.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:44.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:44.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:44.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:44.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:44.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:44.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:44.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:44.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:44.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:45.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:45.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:45.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:45.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:45.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:45.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:45.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:45.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:45.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:45.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:46.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:46.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:46.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:46.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:46.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:46.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:46.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:46.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:46.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:46.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:47.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:47.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:47.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:47.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:47.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:47.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:47.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:48.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:48.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:48.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:48.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:48.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-02-05T05:57:48.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-02-05T05:57:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-02-05T05:57:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-02-05T05:57:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T05:57:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-02-05T05:57:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:57:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T05:57:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T05:57:48.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:57:48.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T05:57:48.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-02-05T05:57:48.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T05:57:49.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-02-05T05:57:49.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-02-05T05:57:49.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-02-05T05:57:49.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-02-05T05:57:49.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-02-05T05:57:49.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:57:49.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-02-05T05:57:49.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-02-05T05:57:49.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-02-05T05:57:49.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T05:57:49.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-02-05T05:57:49.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:57:49.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-02-05T05:57:49.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T05:57:49.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:57:49.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:57:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:57:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:57:49.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:57:50.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:57:50.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-02-05T05:57:50.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:57:50.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:57:50.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:57:50.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:57:50.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:57:50.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:57:50.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-02-05T05:57:50.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:50.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-02-05T05:57:50.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:57:50.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-02-05T05:57:50.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:50.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:50.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:57:50.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:50.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:57:50.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-02-05T05:57:51.555 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-02-05T05:57:52.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:52.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:57:52.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-02-05T05:57:53.171 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:104778fc:::obj2:head# 2024-02-05T05:57:53.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:53.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-02-05T05:57:53.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:57:53.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-02-05T05:57:53.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:53.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:53.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:57:53.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:53.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:57:53.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-02-05T05:57:54.528 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-02-05T05:57:55.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:55.044 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:57:55.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-02-05T05:57:55.789 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-02-05T05:57:56.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:56.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-02-05T05:57:56.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:57:56.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-02-05T05:57:56.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:56.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:56.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:57:56.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:56.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:57:56.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-02-05T05:57:56.959 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:c52c9666:::obj5:head# 2024-02-05T05:57:57.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:57.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:57:57.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-02-05T05:57:58.271 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:233a42c1:::obj6:head# 2024-02-05T05:57:58.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:58.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:57:58.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-02-05T05:57:59.447 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:9a09113a:::obj7:head# 2024-02-05T05:57:59.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:57:59.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:57:59.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-02-05T05:58:00.641 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:a557efb1:::obj8:head# 2024-02-05T05:58:01.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:01.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:01.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-02-05T05:58:01.801 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:fe351e27:::obj9:head# 2024-02-05T05:58:02.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:02.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:02.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-02-05T05:58:02.946 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:d1337354:::obj10:head# 2024-02-05T05:58:03.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:03.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:03.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-02-05T05:58:04.223 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-02-05T05:58:04.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:04.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:04.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-02-05T05:58:05.668 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-02-05T05:58:06.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:06.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-02-05T05:58:06.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:06.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-02-05T05:58:06.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:06.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:06.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:06.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:06.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:06.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-02-05T05:58:07.055 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-02-05T05:58:07.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:07.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-02-05T05:58:08.290 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:e0a44829:::obj14:head# 2024-02-05T05:58:08.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:08.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:08.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-02-05T05:58:09.392 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:ab946124:::obj15:head# 2024-02-05T05:58:09.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:09.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:09.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-02-05T05:58:10.477 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-02-05T05:58:10.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:10.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-02-05T05:58:10.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:10.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-02-05T05:58:10.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:10.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:10.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:10.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:10.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:10.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-02-05T05:58:11.620 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-02-05T05:58:12.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:12.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:12.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-02-05T05:58:12.789 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:b9836a99:::obj18:head# 2024-02-05T05:58:13.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:13.305 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-02-05T05:58:13.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:13.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-02-05T05:58:13.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:13.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:13.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:13.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:13.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:13.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-02-05T05:58:13.966 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-02-05T05:58:14.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:14.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:14.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-02-05T05:58:15.051 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:24b5611b:::obj20:head# 2024-02-05T05:58:15.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:15.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-02-05T05:58:15.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:15.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-02-05T05:58:15.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:15.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:15.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:15.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:15.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:15.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-02-05T05:58:16.346 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-02-05T05:58:16.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:16.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-02-05T05:58:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-02-05T05:58:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:16.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:16.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:16.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:16.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:16.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-02-05T05:58:17.564 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:c0c49def:::obj22:head# 2024-02-05T05:58:18.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:18.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-02-05T05:58:18.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:18.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-02-05T05:58:18.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:18.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:18.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:18.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:18.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:18.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-02-05T05:58:18.750 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:242c40e0:::obj23:head# 2024-02-05T05:58:19.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:19.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-02-05T05:58:19.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:19.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-02-05T05:58:19.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:19.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:19.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:19.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:19.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:19.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-02-05T05:58:19.969 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:7eacede5:::obj24:head# 2024-02-05T05:58:20.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:20.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:20.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-02-05T05:58:21.079 INFO:tasks.workunit.client.0.smithi188.stdout:remove 0#1:7109478f:::obj25:head# 2024-02-05T05:58:21.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-02-05T05:58:21.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-02-05T05:58:21.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-02-05T05:58:21.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-02-05T05:58:21.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T05:58:21.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T05:58:21.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T05:58:21.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T05:58:21.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:21.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-02-05T05:58:22.340 INFO:tasks.workunit.client.0.smithi188.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-02-05T05:58:22.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:58:22.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:58:22.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:58:22.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:58:22.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:58:22.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:58:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:58:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:58:22.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T05:58:22.861 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T05:58:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T05:58:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:58:22.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-02-05T05:58:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T05:58:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:58:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:58:22.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:58:22.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:22.910+0000 7f2b326b24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:22.892 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:22.913+0000 7f2b326b24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:22.893 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:22.914+0000 7f2b326b24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:23.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T05:58:23.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:58:23.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T05:58:23.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:58:23.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:58:23.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:23.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:58:23.138 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:58:23.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:23.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:23.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:23.411 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:23.432+0000 7f2b326b24c0 -1 Falling back to public interface 2024-02-05T05:58:24.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:24.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:24.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:58:24.407 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:58:24.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:24.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:24.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:25.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:25.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:25.679 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:58:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:58:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:25.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:25.694 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:25.715+0000 7f2b326b24c0 -1 osd.1 22 log_to_monitors true 2024-02-05T05:58:25.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:26.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:26.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:26.999 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:58:27.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:58:27.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:27.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T05:58:27.269 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/3062123162,v1:127.0.0.1:6803/3062123162] [v2:127.0.0.1:6804/3062123162,v1:127.0.0.1:6805/3062123162] exists,up c25ce3a7-548b-4db6-8195-bd882f822f2f 2024-02-05T05:58:27.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:58:27.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:58:27.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:58:27.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:58:27.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T05:58:27.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T05:58:27.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T05:58:27.274 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:58:27.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T05:58:27.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-02-05T05:58:27.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:58:27.276 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:58:27.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T05:58:27.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-02-05T05:58:27.277 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T05:58:27.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T05:58:27.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T05:58:27.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T05:58:27.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T05:58:27.299 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:27.320+0000 7f79d77be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:27.301 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:27.322+0000 7f79d77be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:27.302 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:27.323+0000 7f79d77be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:27.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:58:27.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:27.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:27.825 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:27.846+0000 7f79d77be4c0 -1 Falling back to public interface 2024-02-05T05:58:27.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:28.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:28.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:28.883 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:58:28.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:58:28.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:28.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:29.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:30.159 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:58:30.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:58:30.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:30.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:30.225 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:30.246+0000 7f79d77be4c0 -1 osd.0 21 log_to_monitors true 2024-02-05T05:58:30.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:58:31.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:58:31.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:31.532 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:58:31.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:58:31.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:31.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:58:31.810 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/2983692445,v1:127.0.0.1:6811/2983692445] [v2:127.0.0.1:6812/2983692445,v1:127.0.0.1:6813/2983692445] exists,up 792b9be4-8ee3-409e-b2b8-e73459e8119e 2024-02-05T05:58:31.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:58:31.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:58:31.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:58:31.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:58:31.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:58:31.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:58:31.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:58:31.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:58:31.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:58:32.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:58:32.187 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:58:32.187 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:58:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:58:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:32.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:58:32.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T05:58:32.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T05:58:32.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-02-05T05:58:32.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:32.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:58:32.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-02-05T05:58:32.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-02-05T05:58:32.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-02-05T05:58:32.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:32.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:58:32.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-02-05T05:58:32.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-02-05T05:58:32.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509459' 2024-02-05T05:58:32.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:32.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-02-05T05:58:32.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:32.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:58:32.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-02-05T05:58:32.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:32.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T05:58:32.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-02-05T05:58:32.541 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 124554051586 2024-02-05T05:58:32.541 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:32.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-02-05T05:58:32.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:58:33.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:58:33.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:34.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T05:58:34.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:34.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149699 2024-02-05T05:58:34.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:34.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:58:34.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149699 2024-02-05T05:58:34.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:34.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-02-05T05:58:34.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-02-05T05:58:34.094 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 111669149699 2024-02-05T05:58:34.094 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:58:34.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-02-05T05:58:34.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:34.365 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-02-05T05:58:34.365 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:34.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:58:34.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-02-05T05:58:34.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:34.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-02-05T05:58:34.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-02-05T05:58:34.369 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509459 2024-02-05T05:58:34.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:34.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-02-05T05:58:34.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:58:34.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:34.636 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:34.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:58:34.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:34.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:34.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:34.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:34.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:34.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:34.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:35.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:58:35.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:35.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:35.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:58:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:58:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:58:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-02-05T05:58:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T05:58:35.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T05:58:35.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:58:35.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:58:35.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:58:35.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:58:35.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:35.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T05:58:36.072 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:58:36.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:58:36.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:58:36.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:35.805819+0000 '>' 2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:36.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:58:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:58:37.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:58:37.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:58:37.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:58:37.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:58:37.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:58:37.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:58:37.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:35.805819+0000 '>' 2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:37.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:58:38.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:58:38.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:58:38.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:58:38.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:58:38.619 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:58:38.620 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:58:38.620 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:58:38.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:35.805819+0000 '>' 2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:38.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:58:39.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:58:39.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:58:39.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:58:39.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:58:39.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:58:39.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:58:39.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:58:40.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:35.805819+0000 '>' 2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:40.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:58:41.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:58:41.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:58:41.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:58:41.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:58:41.172 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:58:41.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:58:41.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:58:41.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:57:35.805819+0000 '>' 2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:41.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:58:42.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:58:42.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:58:42.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:58:42.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:58:42.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:58:42.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:58:42.446 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:58:42.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:58:36.622325+0000 '>' 2024-02-05T05:57:35.805819+0000 2024-02-05T05:58:42.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:58:42.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-02-05T05:58:42.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:58:42.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:58:42.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:58:42.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:58:42.720 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:58:42.720 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:58:42.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:58:42.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:58:42.720 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:58:42.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:58:42.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:58:42.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:58:42.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:58:42.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:58:42.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:58:43.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:58:43.073 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:58:43.073 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:58:43.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:58:43.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:43.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:58:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-02-05T05:58:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-02-05T05:58:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-02-05T05:58:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:43.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:58:43.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-02-05T05:58:43.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-02-05T05:58:43.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702' 2024-02-05T05:58:43.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:43.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:58:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-02-05T05:58:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-02-05T05:58:43.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702 2-64424509462' 2024-02-05T05:58:43.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-02-05T05:58:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:43.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:58:43.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-02-05T05:58:43.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-02-05T05:58:43.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-02-05T05:58:43.399 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 124554051589 2024-02-05T05:58:43.400 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:43.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-02-05T05:58:43.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:58:44.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:58:44.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:44.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-02-05T05:58:44.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:44.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149702 2024-02-05T05:58:44.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:44.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:58:44.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149702 2024-02-05T05:58:44.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:44.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-02-05T05:58:44.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149702' 2024-02-05T05:58:44.946 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 111669149702 2024-02-05T05:58:44.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:58:45.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-02-05T05:58:45.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:45.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-02-05T05:58:45.217 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:45.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:58:45.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-02-05T05:58:45.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:45.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-02-05T05:58:45.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-02-05T05:58:45.220 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509462 2024-02-05T05:58:45.220 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:45.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-02-05T05:58:45.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:58:45.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:45.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:45.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T05:58:45.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:58:45.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:58:45.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:58:45.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:58:45.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:58:45.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:58:45.825 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:58:46.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T05:58:46.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:58:46.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:58:46.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:58:46.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T05:58:46.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:58:46.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:58:46.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-02-05T05:58:46.688 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T05:58:46.689 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-02-05T05:58:37.948995+0000 21'30 30:128 [1,0,2] 1 [1,0,2] 1 21'30 2024-02-05T05:58:36.622325+0000 21'30 2024-02-05T05:58:36.622325+0000 0 1 periodic scrub scheduled @ 2024-02-06T05:58:36.622325+0000 17 0 2024-02-05T05:58:46.689 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T05:58:46.689 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T05:58:46.689 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T05:58:46.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-02-05T05:58:46.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:58:46.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:58:46.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T05:58:46.968 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T05:58:46.968 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T05:58:46.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:58:46.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:46.968 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:58:47.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-02-05T05:58:47.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-02-05T05:58:47.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-02-05T05:58:47.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:47.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T05:58:47.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149704 2024-02-05T05:58:47.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149704 2024-02-05T05:58:47.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2024-02-05T05:58:47.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:58:47.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T05:58:47.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-02-05T05:58:47.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-02-05T05:58:47.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509464' 2024-02-05T05:58:47.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:47.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-02-05T05:58:47.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:47.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:58:47.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-02-05T05:58:47.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:47.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-02-05T05:58:47.299 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 124554051591 2024-02-05T05:58:47.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-02-05T05:58:47.300 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:47.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051591 2024-02-05T05:58:47.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:58:48.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:58:48.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:58:48.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-02-05T05:58:48.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:48.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149704 2024-02-05T05:58:48.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:48.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T05:58:48.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149704 2024-02-05T05:58:48.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:48.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149704 2024-02-05T05:58:48.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2024-02-05T05:58:48.841 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 111669149704 2024-02-05T05:58:48.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T05:58:49.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149704 2024-02-05T05:58:49.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:58:49.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-02-05T05:58:49.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:58:49.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T05:58:49.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-02-05T05:58:49.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:58:49.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-02-05T05:58:49.110 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509464 2024-02-05T05:58:49.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-02-05T05:58:49.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T05:58:49.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-02-05T05:58:49.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-02-05T05:58:49.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout: "num_bytes": 210, 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects": 30, 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout: "num_object_clones": 0, 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout: "num_object_copies": 90, 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_missing_on_primary": 0, 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_missing": 0, 2024-02-05T05:58:49.482 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_degraded": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_misplaced": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_unfound": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_dirty": 30, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_whiteouts": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_read": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_read_kb": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_write": 30, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_write_kb": 30, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrub_errors": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shallow_scrub_errors": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_deep_scrub_errors": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_recovered": 26, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_bytes_recovered": 182, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_keys_recovered": 0, 2024-02-05T05:58:49.483 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_omap": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_hit_set_archive": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_bytes_hit_set_archive": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush_kb": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict_kb": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_promote": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush_mode_high": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_flush_mode_low": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict_mode_some": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_evict_mode_full": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_pinned": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_legacy_snapsets": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_large_omap_objects": 0, 2024-02-05T05:58:49.484 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_manifest": 0, 2024-02-05T05:58:49.485 INFO:tasks.workunit.client.0.smithi188.stdout: "num_omap_bytes": 0, 2024-02-05T05:58:49.485 INFO:tasks.workunit.client.0.smithi188.stdout: "num_omap_keys": 0, 2024-02-05T05:58:49.485 INFO:tasks.workunit.client.0.smithi188.stdout: "num_objects_repaired": 26 2024-02-05T05:58:49.485 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:58:49.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-02-05T05:58:49.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-02-05T05:58:49.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-02-05T05:58:49.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-02-05T05:58:49.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-02-05T05:58:49.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-02-05T05:58:49.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:58:49.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-02-05T05:58:49.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-02-05T05:58:49.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-02-05T05:58:49.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:49.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-02-05T05:58:49.843 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:49.854 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:58:49.854 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T05:58:49.854 INFO:tasks.workunit.client.0.smithi188.stdout: "up_from": 29, 2024-02-05T05:58:49.854 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 124554051591, 2024-02-05T05:58:49.854 INFO:tasks.workunit.client.0.smithi188.stdout: "num_pgs": 1, 2024-02-05T05:58:49.854 INFO:tasks.workunit.client.0.smithi188.stdout: "num_osds": 1, 2024-02-05T05:58:49.854 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_osds": 1, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "kb": 104857600, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used": 28472, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_data": 368, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_omap": 4, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_meta": 28091, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_avail": 104829128, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "statfs": { 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "total": 107374182400, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "available": 107345027072, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "internally_reserved": 0, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "allocated": 376832, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "data_stored": 201934, 2024-02-05T05:58:49.855 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed": 0, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_allocated": 0, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_original": 0, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_allocated": 4402, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "internal_metadata": 28765902 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "hb_peers": [ 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: 1, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: 2 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "snap_trim_queue_len": 0, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "num_snap_trimming": 0, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shards_repaired": 13, 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "op_queue_age_hist": { 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "histogram": [], 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: "upper_bound": 1 2024-02-05T05:58:49.856 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "perf_stat": { 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ms": 0, 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ms": 0, 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ns": 0, 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ns": 0 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "alerts": [], 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "network_ping_times": [ 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:58:49.857 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.189 2024-02-05T05:58:49.858 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 2, 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:58:49.859 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.245 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:49.860 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:58:49.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:49.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-02-05T05:58:50.109 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:50.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-02-05T05:58:50.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-02-05T05:58:50.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:58:50.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2024-02-05T05:58:50.122 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-02-05T05:58:50.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-02-05T05:58:50.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:50.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-02-05T05:58:50.376 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:50.387 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:58:50.387 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "up_from": 26, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 111669149704, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "num_pgs": 1, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "num_osds": 1, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_osds": 1, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "kb": 104857600, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used": 28472, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_data": 368, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_omap": 4, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_meta": 28091, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_avail": 104829128, 2024-02-05T05:58:50.388 INFO:tasks.workunit.client.0.smithi188.stdout: "statfs": { 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "total": 107374182400, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "available": 107345027072, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "internally_reserved": 0, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "allocated": 376832, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "data_stored": 201934, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed": 0, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_allocated": 0, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_original": 0, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_allocated": 4398, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "internal_metadata": 28765906 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: "hb_peers": [ 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: 0, 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: 2 2024-02-05T05:58:50.389 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "snap_trim_queue_len": 0, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "num_snap_trimming": 0, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shards_repaired": 13, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "op_queue_age_hist": { 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "histogram": [], 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "upper_bound": 1 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "perf_stat": { 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ms": 0, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ms": 0, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ns": 0, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ns": 0 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.390 INFO:tasks.workunit.client.0.smithi188.stdout: "alerts": [], 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "network_ping_times": [ 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:58:50.391 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.224 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.392 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 2, 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.393 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.26800000000000002 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:50.394 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:58:50.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:50.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-02-05T05:58:50.636 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=13 2024-02-05T05:58:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 13 = 13 2024-02-05T05:58:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T05:58:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2024-02-05T05:58:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: repair=0 2024-02-05T05:58:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:50.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2024-02-05T05:58:50.900 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:50.911 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:58:50.911 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 2, 2024-02-05T05:58:50.911 INFO:tasks.workunit.client.0.smithi188.stdout: "up_from": 15, 2024-02-05T05:58:50.911 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 64424509464, 2024-02-05T05:58:50.911 INFO:tasks.workunit.client.0.smithi188.stdout: "num_pgs": 1, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "num_osds": 1, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_osds": 1, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_omap_osds": 1, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "kb": 104857600, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used": 27204, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_data": 368, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_omap": 1, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_meta": 26814, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_avail": 104830396, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "statfs": { 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "total": 107374182400, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "available": 107346325504, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "internally_reserved": 0, 2024-02-05T05:58:50.912 INFO:tasks.workunit.client.0.smithi188.stdout: "allocated": 376832, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "data_stored": 201934, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed": 0, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_allocated": 0, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_original": 0, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_allocated": 1595, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "internal_metadata": 27457989 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "hb_peers": [ 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: 0, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "snap_trim_queue_len": 0, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "num_snap_trimming": 0, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shards_repaired": 0, 2024-02-05T05:58:50.913 INFO:tasks.workunit.client.0.smithi188.stdout: "op_queue_age_hist": { 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "histogram": [], 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "upper_bound": 1 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "perf_stat": { 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ms": 0, 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ms": 0, 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ns": 0, 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ns": 0 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "alerts": [], 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "network_ping_times": [ 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:58:50.914 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.915 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.22700000000000001 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "last update": "Thu Jan 1 00:00:00 1970", 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "interfaces": [ 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "interface": "back", 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "average": { 2024-02-05T05:58:50.916 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "min": { 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "max": { 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "1min": 0, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "5min": 0, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "15min": 0 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: "last": 0.25700000000000001 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:50.917 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:50.918 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T05:58:50.918 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T05:58:50.918 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:58:50.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:50.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2024-02-05T05:58:51.165 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:51.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=0 2024-02-05T05:58:51.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 0 = 0 2024-02-05T05:58:51.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:51.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2024-02-05T05:58:51.426 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:51.437 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:58:51.437 INFO:tasks.workunit.client.0.smithi188.stdout: "up_from": 0, 2024-02-05T05:58:51.437 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T05:58:51.437 INFO:tasks.workunit.client.0.smithi188.stdout: "num_pgs": 3, 2024-02-05T05:58:51.437 INFO:tasks.workunit.client.0.smithi188.stdout: "num_osds": 3, 2024-02-05T05:58:51.437 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_osds": 3, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "num_per_pool_omap_osds": 3, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "kb": 314572800, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used": 84148, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_data": 1104, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_omap": 10, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_used_meta": 82997, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "kb_avail": 314488652, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "statfs": { 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "total": 322122547200, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "available": 322036379648, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "internally_reserved": 0, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "allocated": 1130496, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "data_stored": 605802, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed": 0, 2024-02-05T05:58:51.438 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_allocated": 0, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "data_compressed_original": 0, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_allocated": 10395, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "internal_metadata": 84989797 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "hb_peers": [], 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "snap_trim_queue_len": 0, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "num_snap_trimming": 0, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "num_shards_repaired": 26, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "op_queue_age_hist": { 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "histogram": [], 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "upper_bound": 1 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "perf_stat": { 2024-02-05T05:58:51.439 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ms": 0, 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ms": 0, 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stdout: "commit_latency_ns": 0, 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stdout: "apply_latency_ns": 0 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stdout: "alerts": [], 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stdout: "network_ping_times": [] 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:58:51.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-02-05T05:58:51.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-02-05T05:58:51.687 INFO:tasks.workunit.client.0.smithi188.stderr:dumped all 2024-02-05T05:58:51.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: COUNT=26 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:794: TEST_repair_stats_ec: test 26 = 26 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:58:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:58:51.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:58:51.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:58:51.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:58:51.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:58:51.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:58:51.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:58:51.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:58:51.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:58:51.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:58:51.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:58:51.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:58:51.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:58:51.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:58:51.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:58:51.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:58:51.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:51.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:51.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:58:51.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:58:51.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T05:58:51.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T05:58:51.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T05:58:51.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T05:58:51.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T05:58:51.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T05:58:51.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T05:58:51.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T05:58:51.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T05:58:51.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:58:51.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T05:58:51.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T05:58:51.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T05:58:51.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T05:58:51.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T05:58:51.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T05:58:51.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T05:58:51.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:51.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:51.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T05:58:51.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T05:58:51.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T05:58:51.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T05:58:51.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T05:58:51.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:51.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:51.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T05:58:51.849 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T05:58:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T05:58:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T05:58:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T05:58:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2024-02-05T05:58:51.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6197: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6198: TEST_request_scrub_priority: local poolname=psr_pool 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6199: TEST_request_scrub_priority: local objname=POBJ 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6200: TEST_request_scrub_priority: local OBJECTS=64 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6201: TEST_request_scrub_priority: local PGS=8 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6203: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T05:58:51.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T05:58:52.257 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T05:58:52.257 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:52.257 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:52.257 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:52.258 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:52.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:52.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T05:58:52.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T05:58:52.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T05:58:52.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:58:52.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:58:52.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T05:58:52.451 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T05:58:52.452 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:58:52.452 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:58:52.452 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:58:52.452 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:58:52.452 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:52.453 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:52.453 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:58:52.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:58:52.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T05:58:52.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T05:58:52.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T05:58:52.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6204: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2024-02-05T05:58:52.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T05:58:52.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T05:58:52.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T05:58:52.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T05:58:52.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T05:58:52.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T05:58:53.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T05:58:53.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:53.419 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:53.419 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:53.419 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:53.420 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:53.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:53.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:58:53.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T05:58:53.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6205: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-02-05T05:58:53.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6206: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-02-05T05:58:53.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6207: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:58:53.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T05:58:53.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T05:58:53.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T05:58:53.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T05:58:53.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T05:58:53.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T05:58:53.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T05:58:53.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T05:58:53.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T05:58:53.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T05:58:53.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T05:58:53.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T05:58:53.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T05:58:53.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T05:58:53.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T05:58:53.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T05:58:53.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T05:58:53.458 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T05:58:53.459 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T05:58:53.459 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T05:58:53.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:53.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T05:58:53.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T05:58:53.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T05:58:53.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T05:58:53.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T05:58:53.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T05:58:53.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T05:58:53.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T05:58:53.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T05:58:53.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T05:58:53.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T05:58:53.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-02-05T05:58:53.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T05:58:53.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T05:58:53.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3893a261-5a6a-49e4-b553-77cec1a4ecfb 2024-02-05T05:58:53.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3893a261-5a6a-49e4-b553-77cec1a4ecfb' 2024-02-05T05:58:53.462 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 3893a261-5a6a-49e4-b553-77cec1a4ecfb 2024-02-05T05:58:53.463 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T05:58:53.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdecBlFi85HRAAOeq0SZLD+GbjRu0Oc5/1Nw== 2024-02-05T05:58:53.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdecBlFi85HRAAOeq0SZLD+GbjRu0Oc5/1Nw=="}' 2024-02-05T05:58:53.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3893a261-5a6a-49e4-b553-77cec1a4ecfb -i td/osd-scrub-repair/0/new.json 2024-02-05T05:58:53.675 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:58:53.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T05:58:53.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQAdecBlFi85HRAAOeq0SZLD+GbjRu0Oc5/1Nw== --osd-uuid 3893a261-5a6a-49e4-b553-77cec1a4ecfb 2024-02-05T05:58:53.717 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:53.738+0000 7f766a8154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:53.726 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:53.747+0000 7f766a8154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:53.727 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:53.749+0000 7f766a8154c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:53.728 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:53.749+0000 7f766a8154c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:58:53.728 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:53.749+0000 7f766a8154c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:58:53.728 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:53.749+0000 7f766a8154c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T05:58:53.818 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:53.839+0000 7f766a8154c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T05:58:59.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T05:58:59.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T05:58:59.191 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T05:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T05:58:59.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T05:58:59.553 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T05:58:59.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T05:58:59.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-02-05T05:58:59.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T05:58:59.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T05:58:59.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T05:58:59.581 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:59.603+0000 7f14e2cb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:59.583 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:59.605+0000 7f14e2cb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:59.584 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:58:59.606+0000 7f14e2cb44c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T05:58:59.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T05:58:59.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T05:58:59.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T05:58:59.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T05:58:59.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T05:58:59.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:58:59.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T05:58:59.826 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T05:58:59.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:58:59.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:00.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:00.111 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:59:00.133+0000 7f14e2cb44c0 -1 Falling back to public interface 2024-02-05T05:59:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:01.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:01.093 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T05:59:01.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T05:59:01.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:01.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:01.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:02.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:02.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:02.371 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T05:59:02.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T05:59:02.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:02.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:02.582 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T05:59:02.603+0000 7f14e2cb44c0 -1 osd.0 0 log_to_monitors true 2024-02-05T05:59:02.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:03.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:03.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:03.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T05:59:03.684 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T05:59:03.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:03.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:03.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:04.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:04.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:04.960 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T05:59:04.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T05:59:04.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:04.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:05.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:06.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:06.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:06.336 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T05:59:06.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T05:59:06.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:06.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:06.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:07.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:07.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:07.614 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T05:59:07.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T05:59:07.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:07.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:07.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:08.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:08.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:08.894 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T05:59:08.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T05:59:08.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:08.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:09.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:10.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:10.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:10.179 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T05:59:10.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T05:59:10.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:10.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:10.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:11.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:11.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:11.538 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T05:59:11.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T05:59:11.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:11.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:11.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:12.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:12.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:12.816 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T05:59:12.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T05:59:12.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:12.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:13.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:14.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:14.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:14.165 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T05:59:14.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T05:59:14.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:14.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:14.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:15.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:15.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:15.490 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T05:59:15.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T05:59:15.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:15.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:15.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:16.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:16.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:16.762 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T05:59:16.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T05:59:16.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:16.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:17.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:18.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:18.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:18.139 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T05:59:18.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T05:59:18.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:18.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:18.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:19.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:19.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:19.432 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T05:59:19.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T05:59:19.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:19.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:19.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:20.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:20.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:20.726 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T05:59:20.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T05:59:20.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:20.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:21.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:22.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:22.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:22.221 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T05:59:22.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T05:59:22.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:22.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:22.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:23.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:23.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:23.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T05:59:23.559 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T05:59:23.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:23.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:24.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:25.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:25.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:25.175 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T05:59:25.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T05:59:25.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:25.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:25.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T05:59:26.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T05:59:26.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T05:59:26.512 INFO:tasks.workunit.client.0.smithi188.stdout:20 2024-02-05T05:59:26.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-02-05T05:59:26.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T05:59:26.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T05:59:26.787 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1927912673,v1:127.0.0.1:6803/1927912673] [v2:127.0.0.1:6804/1927912673,v1:127.0.0.1:6805/1927912673] exists,up 3893a261-5a6a-49e4-b553-77cec1a4ecfb 2024-02-05T05:59:26.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T05:59:26.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T05:59:26.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T05:59:26.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6209: TEST_request_scrub_priority: create_pool psr_pool 8 8 2024-02-05T05:59:26.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 8 8 2024-02-05T05:59:27.151 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool' created 2024-02-05T05:59:27.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T05:59:28.164 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6210: TEST_request_scrub_priority: wait_for_clean 2024-02-05T05:59:28.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T05:59:28.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T05:59:28.165 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T05:59:28.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T05:59:28.165 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T05:59:28.165 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T05:59:28.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T05:59:28.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T05:59:28.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T05:59:28.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T05:59:28.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T05:59:28.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T05:59:28.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T05:59:28.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:59:28.267 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:59:28.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T05:59:28.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:59:28.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:28.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:59:28.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-02-05T05:59:28.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-02-05T05:59:28.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-02-05T05:59:28.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:28.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-02-05T05:59:28.642 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:28.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:59:28.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-02-05T05:59:28.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:28.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-02-05T05:59:28.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-02-05T05:59:28.645 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836489 2024-02-05T05:59:28.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:28.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-02-05T05:59:28.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T05:59:29.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T05:59:29.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:30.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-02-05T05:59:30.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T05:59:30.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:30.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:30.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-02-05T05:59:30.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T05:59:30.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T05:59:30.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T05:59:30.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T05:59:30.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T05:59:30.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T05:59:30.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T05:59:30.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-02-05T05:59:30.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T05:59:30.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T05:59:30.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T05:59:31.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6212: TEST_request_scrub_priority: local osd=0 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6213: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T05:59:31.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T05:59:31.662 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T05:59:31.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T05:59:32.181 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T05:59:32.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T05:59:32.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T05:59:32.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-02-05T05:59:32.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6214: TEST_request_scrub_priority: get_primary psr_pool POBJ 2024-02-05T05:59:32.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-02-05T05:59:32.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-02-05T05:59:32.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-02-05T05:59:32.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T05:59:32.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6214: TEST_request_scrub_priority: local primary=0 2024-02-05T05:59:32.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: get_pg psr_pool POBJ 2024-02-05T05:59:32.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-02-05T05:59:32.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-02-05T05:59:32.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-02-05T05:59:32.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T05:59:32.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6215: TEST_request_scrub_priority: local pg=1.1 2024-02-05T05:59:32.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: ceph osd dump 2024-02-05T05:59:32.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2024-02-05T05:59:32.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: awk '{ print $2 }' 2024-02-05T05:59:33.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6216: TEST_request_scrub_priority: poolid=1 2024-02-05T05:59:33.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6218: TEST_request_scrub_priority: local otherpgs 2024-02-05T05:59:33.263 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: expr 8 - 1 2024-02-05T05:59:33.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: seq 0 7 2024-02-05T05:59:33.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:33.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.0 2024-02-05T05:59:33.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2024-02-05T05:59:33.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 ' 2024-02-05T05:59:33.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:33.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:33.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:33.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:33.266 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:33.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:33.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2024-02-05T05:59:33.689 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:33.689 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:33.689 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:59:33.689 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:57:53.711913+0000" 2024-02-05T05:59:33.689 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:33.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:33.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.1 2024-02-05T05:59:33.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2024-02-05T05:59:33.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6223: TEST_request_scrub_priority: continue 2024-02-05T05:59:33.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:33.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.2 2024-02-05T05:59:33.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2024-02-05T05:59:33.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2024-02-05T05:59:33.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:33.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:33.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:33.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:33.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:33.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:33.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2024-02-05T05:59:34.059 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:34.059 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:34.059 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:59:34.059 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:57:54.082067+0000" 2024-02-05T05:59:34.059 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:34.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:34.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.3 2024-02-05T05:59:34.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2024-02-05T05:59:34.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2024-02-05T05:59:34.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:34.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:34.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:34.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:34.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:34.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:34.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2024-02-05T05:59:34.427 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:34.428 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:34.428 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:59:34.428 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:57:54.450303+0000" 2024-02-05T05:59:34.428 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.4 2024-02-05T05:59:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2024-02-05T05:59:34.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2024-02-05T05:59:34.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:34.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:34.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:34.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:34.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:34.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:34.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2024-02-05T05:59:34.812 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:34.812 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:34.812 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:59:34.812 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:57:54.834615+0000" 2024-02-05T05:59:34.812 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:34.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:34.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.5 2024-02-05T05:59:34.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2024-02-05T05:59:34.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2024-02-05T05:59:34.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:34.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:34.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:34.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:34.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:35.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:35.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2024-02-05T05:59:35.181 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:35.182 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:35.182 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:59:35.182 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:57:55.204436+0000" 2024-02-05T05:59:35.182 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:35.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:35.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.6 2024-02-05T05:59:35.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2024-02-05T05:59:35.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2024-02-05T05:59:35.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:35.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:35.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:35.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:35.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:35.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:35.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2024-02-05T05:59:35.553 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:35.553 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:35.553 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:59:35.553 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:57:55.575634+0000" 2024-02-05T05:59:35.553 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:35.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6219: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2024-02-05T05:59:35.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6221: TEST_request_scrub_priority: opg=1.7 2024-02-05T05:59:35.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6222: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2024-02-05T05:59:35.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6225: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2024-02-05T05:59:35.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:35.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:35.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:35.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:35.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:35.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6226: TEST_request_scrub_priority: local other_last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:35.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6228: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2024-02-05T05:59:35.924 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:35.924 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:35.925 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T05:59:35.925 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T05:57:55.947452+0000" 2024-02-05T05:59:35.925 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:35.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6231: TEST_request_scrub_priority: sleep 15 2024-02-05T05:59:50.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6232: TEST_request_scrub_priority: flush_pg_stats 2024-02-05T05:59:50.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T05:59:50.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T05:59:51.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T05:59:51.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T05:59:51.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T05:59:51.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T05:59:51.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-02-05T05:59:51.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-02-05T05:59:51.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-02-05T05:59:51.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T05:59:51.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-02-05T05:59:51.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T05:59:51.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T05:59:51.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-02-05T05:59:51.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T05:59:51.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-02-05T05:59:51.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-02-05T05:59:51.317 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836495 2024-02-05T05:59:51.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T05:59:51.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-02-05T05:59:51.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2024-02-05T05:59:51.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:51.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:51.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:51.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:51.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6235: TEST_request_scrub_priority: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:51.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6236: TEST_request_scrub_priority: ceph tell 1.1 scrub 2024-02-05T05:59:51.970 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T05:59:51.970 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T05:59:51.970 INFO:tasks.workunit.client.0.smithi188.stdout: "must": true, 2024-02-05T05:59:51.970 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "0.000000" 2024-02-05T05:59:51.970 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T05:59:51.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6238: TEST_request_scrub_priority: ceph osd unset noscrub 2024-02-05T05:59:52.337 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is unset 2024-02-05T05:59:52.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6239: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2024-02-05T05:59:52.856 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is unset 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6241: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.1 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:52.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:52.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:52.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:53.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:27.125039+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:53.134 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:59:54.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:59:54.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:54.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T05:59:54.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:54.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:54.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:54.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:54.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:27.125039+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:54.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:59:55.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:59:55.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:55.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T05:59:55.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:55.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:55.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:55.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:55.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:27.125039+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:55.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:59:56.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:59:56.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:56.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T05:59:56.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:56.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:56.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:56.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:56.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:27.125039+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:56.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T05:59:57.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T05:59:57.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:57.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T05:59:57.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T05:59:57.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:57.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:57.978 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T05:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:52.988650+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T05:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.0 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T05:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:59:58.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:59:58.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:58.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T05:59:58.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T05:59:58.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:58.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:58.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:53.934140+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.2 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.2 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-02-05T05:59:58.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:58.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:58.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-02-05T05:59:58.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:54.923644+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.3 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:58.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T05:59:59.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:55.967325+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:59.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T05:59:59.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T05:59:59.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.4 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:59.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.4 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T05:59:59.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T05:59:59.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T05:57:54.834615+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T05:59:59.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:00:00.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:00:00.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:00.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T06:00:00.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T06:00:00.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:00.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:00.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T06:00:00.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-29T05:57:54.834615+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:00.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:00:01.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:00:01.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:01.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-02-05T06:00:01.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-02-05T06:00:01.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:01.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:01.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-02-05T06:00:01.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:56.939190+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:01.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:00:01.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:00:01.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.5 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.5 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:01.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-02-05T06:00:02.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:57.989587+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:00:02.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:00:02.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.6 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.6 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:02.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-02-05T06:00:02.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:58.957559+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:00:02.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:00:02.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.7 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.7 2024-02-05T06:00:02.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:00:02.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:00:02.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:02.390 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-02-05T06:00:02.390 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-02-05T06:00:02.390 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:02.390 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:02.390 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-02-05T06:00:02.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:59.987959+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6243: TEST_request_scrub_priority: for opg in $otherpgs $pg 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6245: TEST_request_scrub_priority: wait_for_scrub 1.1 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.1 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:00:02.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:00:02.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-02-05T06:00:02.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T05:59:52.988650+0000 '>' 2024-02-05T05:59:27.125039+0000 2024-02-05T06:00:02.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:00:02.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2024-02-05T06:00:02.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep -v purged_snaps 2024-02-05T06:00:02.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: head -1 2024-02-05T06:00:02.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2024-02-05T06:00:02.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6249: TEST_request_scrub_priority: grep -q 1.1 2024-02-05T06:00:02.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:00:02.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:00:02.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:00:02.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:00:02.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:00:02.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:00:02.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:00:02.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:00:02.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:00:03.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:00:03.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:00:03.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:00:03.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:00:03.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:00:03.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:00:03.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:00:03.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:03.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:00:03.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:00:03.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:03.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:00:03.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:00:03.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:00:03.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:00:03.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:03.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:03.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:00:03.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:00:03.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:00:03.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:00:03.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:00:03.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:00:03.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:00:03.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:00:03.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:00:03.042 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:00:03.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:00:03.044 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:00:03.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:00:03.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:00:03.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:00:03.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:03.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:00:03.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:00:03.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:00:03.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:00:03.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:00:03.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:00:03.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:00:03.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:03.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:03.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:00:03.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:00:03.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:00:03.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:00:03.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:00:03.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:03.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:03.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5842: TEST_scrub_warning: local dir=td/osd-scrub-repair 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5843: TEST_scrub_warning: local poolname=psr_pool 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5844: TEST_scrub_warning: local objname=POBJ 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5845: TEST_scrub_warning: local scrubs=5 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5846: TEST_scrub_warning: local deep_scrubs=5 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5847: TEST_scrub_warning: local i1_day=86400 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: calc 86400 '*' 7 2024-02-05T06:00:03.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 7}' 2024-02-05T06:00:03.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5848: TEST_scrub_warning: local i7_days=604800 2024-02-05T06:00:03.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: calc 86400 '*' 14 2024-02-05T06:00:03.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 86400 * 14}' 2024-02-05T06:00:03.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5849: TEST_scrub_warning: local i14_days=1209600 2024-02-05T06:00:03.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5850: TEST_scrub_warning: local overdue=0.5 2024-02-05T06:00:03.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2024-02-05T06:00:03.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2024-02-05T06:00:03.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5851: TEST_scrub_warning: local conf_overdue_seconds=993600 2024-02-05T06:00:03.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2024-02-05T06:00:03.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2024-02-05T06:00:03.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5852: TEST_scrub_warning: local pool_overdue_seconds=1900800 2024-02-05T06:00:03.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5854: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:00:03.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:00:03.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:00:03.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:00:03.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:00:03.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:00:03.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:00:04.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:00:04.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:00:04.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:00:04.132 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:00:04.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:04.133 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:04.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:00:04.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-02-05T06:00:04.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:00:04.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:00:04.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:00:04.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:00:04.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:00:04.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:00:04.318 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:00:04.318 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:00:04.318 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:00:04.319 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:00:04.319 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:04.319 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:04.319 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:00:04.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:00:04.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:00:04.408 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:00:04.409 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:00:04.409 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:04.409 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:04.409 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:00:04.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:00:04.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T06:00:04.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5855: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-02-05T06:00:04.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:00:04.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:00:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:00:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:00:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:00:04.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:00:05.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:00:05.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:00:05.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:00:05.335 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:00:05.335 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:05.335 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:05.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:00:05.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:00:05.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5856: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:00:05.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:00:05.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:00:05.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae77a63a-21dd-453c-94c1-a5c8a87cac91 2024-02-05T06:00:05.372 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 ae77a63a-21dd-453c-94c1-a5c8a87cac91 2024-02-05T06:00:05.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ae77a63a-21dd-453c-94c1-a5c8a87cac91' 2024-02-05T06:00:05.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:00:05.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlecBl/0JoGBAAQZLFHW2xXXGYbwHoDcXX7A== 2024-02-05T06:00:05.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlecBl/0JoGBAAQZLFHW2xXXGYbwHoDcXX7A=="}' 2024-02-05T06:00:05.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae77a63a-21dd-453c-94c1-a5c8a87cac91 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:00:05.590 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:00:05.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:00:05.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQBlecBl/0JoGBAAQZLFHW2xXXGYbwHoDcXX7A== --osd-uuid ae77a63a-21dd-453c-94c1-a5c8a87cac91 2024-02-05T06:00:05.630 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:05.652+0000 7f876ba204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:05.632 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:05.655+0000 7f876ba204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:05.633 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:05.656+0000 7f876ba204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:05.634 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:05.656+0000 7f876ba204c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:00:05.634 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:05.656+0000 7f876ba204c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:00:05.634 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:05.656+0000 7f876ba204c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:00:05.708 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:05.730+0000 7f876ba204c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:00:11.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:00:11.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:00:11.235 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:00:11.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:00:11.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:00:11.582 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:00:11.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:00:11.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2024-02-05T06:00:11.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:00:11.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:00:11.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:00:11.609 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:11.631+0000 7fb7bdc054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:11.611 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:11.633+0000 7fb7bdc054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:11.612 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:11.634+0000 7fb7bdc054c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:00:11.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:00:11.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:00:11.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:00:11.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:00:11.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:00:11.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:11.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:00:11.852 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:00:11.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:11.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:12.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:12.137 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:12.160+0000 7fb7bdc054c0 -1 Falling back to public interface 2024-02-05T06:00:13.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:13.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:13.118 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:00:13.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:00:13.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:13.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:13.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:14.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:14.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:14.397 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:00:14.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:00:14.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:14.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:14.523 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:00:14.545+0000 7fb7bdc054c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:00:14.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:15.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:15.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:15.716 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:00:15.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:00:15.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:15.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:15.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:16.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:16.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:16.993 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:00:16.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:00:16.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:16.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:17.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:18.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:18.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:18.365 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:00:18.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:00:18.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:18.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:18.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:19.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:19.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:19.641 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:00:19.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:00:19.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:19.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:19.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:20.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:20.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:20.922 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:00:20.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:00:20.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:20.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:21.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:22.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:22.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:22.205 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:00:22.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:00:22.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:22.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:22.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:23.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:23.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:23.599 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:00:23.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:00:23.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:23.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:23.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:24.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:24.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:24.877 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:00:24.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:00:24.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:24.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:25.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:26.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:26.206 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:00:26.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:00:26.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:26.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:26.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:27.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:27.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:27.720 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:00:27.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:00:27.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:27.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:28.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:29.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:29.288 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:00:29.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:29.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:00:29.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:29.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:30.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:31.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:31.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:31.236 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:00:31.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:00:31.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:31.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:31.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:32.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:32.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:32.655 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:00:32.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:00:32.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:32.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:33.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:34.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:34.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:34.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:00:34.358 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:00:34.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:34.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:35.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:36.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:36.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:36.094 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:00:36.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:00:36.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:36.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:36.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:00:37.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:00:37.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:00:37.831 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:00:37.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:00:37.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:00:37.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:00:38.106 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3945078658,v1:127.0.0.1:6803/3945078658] [v2:127.0.0.1:6804/3945078658,v1:127.0.0.1:6805/3945078658] exists,up ae77a63a-21dd-453c-94c1-a5c8a87cac91 2024-02-05T06:00:38.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:00:38.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:00:38.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:00:38.107 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: expr 5 + 5 2024-02-05T06:00:38.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: seq 1 10 2024-02-05T06:00:38.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:00:38.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-1 1 1 2024-02-05T06:00:38.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-1 1 1 2024-02-05T06:00:38.474 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-1' created 2024-02-05T06:00:38.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:00:39.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:00:39.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:00:39.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:00:39.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:00:39.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:00:39.489 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:00:39.489 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:00:39.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:00:39.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:00:39.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:00:39.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:00:39.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:00:39.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:00:39.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:00:39.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:00:39.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:00:39.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:00:39.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:00:39.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:39.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:00:39.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836488 2024-02-05T06:00:39.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836488 2024-02-05T06:00:39.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836488' 2024-02-05T06:00:39.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:39.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836488 2024-02-05T06:00:39.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:39.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:00:39.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836488 2024-02-05T06:00:39.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:39.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836488 2024-02-05T06:00:39.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-02-05T06:00:39.963 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836488 2024-02-05T06:00:39.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:40.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836488 2024-02-05T06:00:40.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:00:41.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:00:41.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:41.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836488 2024-02-05T06:00:41.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:00:41.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:41.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:41.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:00:41.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:00:41.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:00:41.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:00:41.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:00:41.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:00:41.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:00:41.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:00:42.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:00:42.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:00:42.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:42.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:42.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:00:42.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:00:42.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:00:42.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 1 = 1 ']' 2024-02-05T06:00:42.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5864: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2024-02-05T06:00:42.811 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 scrub_max_interval to 1209600 2024-02-05T06:00:42.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:00:42.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 1 = 6 ']' 2024-02-05T06:00:42.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:00:42.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-2 1 1 2024-02-05T06:00:42.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-2 1 1 2024-02-05T06:00:43.179 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-2' created 2024-02-05T06:00:43.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:00:44.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:00:44.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:00:44.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:00:44.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:00:44.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:00:44.193 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:00:44.193 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:00:44.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:00:44.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:00:44.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:00:44.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:00:44.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:00:44.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:00:44.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:00:44.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:00:44.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:00:44.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:00:44.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:00:44.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:44.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:00:44.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-02-05T06:00:44.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-02-05T06:00:44.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-02-05T06:00:44.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:44.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-02-05T06:00:44.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:44.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:00:44.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-02-05T06:00:44.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:44.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-02-05T06:00:44.670 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836490 2024-02-05T06:00:44.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-02-05T06:00:44.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:44.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-02-05T06:00:44.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:00:45.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:00:45.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:46.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-02-05T06:00:46.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:00:46.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:46.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:46.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-02-05T06:00:46.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:00:46.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:00:46.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:00:46.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:00:46.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:00:46.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:00:46.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:00:46.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-02-05T06:00:46.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:00:46.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:46.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:47.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-02-05T06:00:47.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:00:47.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:00:47.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 2 = 1 ']' 2024-02-05T06:00:47.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:00:47.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 2 = 6 ']' 2024-02-05T06:00:47.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:00:47.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-3 1 1 2024-02-05T06:00:47.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-3 1 1 2024-02-05T06:00:47.465 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-3' created 2024-02-05T06:00:47.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:00:48.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:00:48.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:00:48.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:00:48.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:00:48.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:00:48.479 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:00:48.479 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:00:48.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:00:48.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:00:48.479 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:00:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:00:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:00:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:00:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:00:48.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:00:48.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:00:48.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:00:48.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:00:48.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:48.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:00:48.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-02-05T06:00:48.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-02-05T06:00:48.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-02-05T06:00:48.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:48.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-02-05T06:00:48.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:48.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:00:48.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-02-05T06:00:48.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-02-05T06:00:48.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-02-05T06:00:48.951 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836492 2024-02-05T06:00:48.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:49.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836492 2024-02-05T06:00:49.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:00:50.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:00:50.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:50.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-02-05T06:00:50.488 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:00:50.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:50.489 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:00:50.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:00:51.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-02-05T06:00:51.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:00:51.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:51.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:51.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-02-05T06:00:51.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:00:51.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:00:51.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 3 = 1 ']' 2024-02-05T06:00:51.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:00:51.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 3 = 6 ']' 2024-02-05T06:00:51.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:00:51.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-4 1 1 2024-02-05T06:00:51.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-4 1 1 2024-02-05T06:00:51.783 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-4' created 2024-02-05T06:00:51.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:00:52.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:00:52.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:00:52.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:00:52.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:00:52.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:00:52.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:00:52.798 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:00:52.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:00:52.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:00:52.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:00:52.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:00:52.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:00:52.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:00:52.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:00:52.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:00:52.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:00:53.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:00:53.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:00:53.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:53.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:00:53.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-02-05T06:00:53.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-02-05T06:00:53.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-02-05T06:00:53.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:53.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-02-05T06:00:53.269 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:53.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:00:53.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-02-05T06:00:53.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:53.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-02-05T06:00:53.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-02-05T06:00:53.273 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836494 2024-02-05T06:00:53.273 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:53.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-02-05T06:00:53.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:00:54.554 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:00:54.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:54.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-02-05T06:00:54.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:00:55.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:00:55.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:56.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-02-05T06:00:56.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:00:56.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:56.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:56.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:00:56.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:00:56.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:00:56.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:00:56.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:00:56.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:00:56.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:00:56.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:00:56.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:00:56.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:00:56.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:00:56.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:00:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:00:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:00:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:00:57.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 4 = 1 ']' 2024-02-05T06:00:57.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:00:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 4 = 6 ']' 2024-02-05T06:00:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:00:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-5 1 1 2024-02-05T06:00:57.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-5 1 1 2024-02-05T06:00:57.521 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-5' created 2024-02-05T06:00:57.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:00:58.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:00:58.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:00:58.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:00:58.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:00:58.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:00:58.535 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:00:58.535 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:00:58.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:00:58.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:00:58.535 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:00:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:00:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:00:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:00:58.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:00:58.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:00:58.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:00:58.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:00:58.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:00:58.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:00:58.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:00:59.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-02-05T06:00:59.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-02-05T06:00:59.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-02-05T06:00:59.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:00:59.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-02-05T06:00:59.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:00:59.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:00:59.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-02-05T06:00:59.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:00:59.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-02-05T06:00:59.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-02-05T06:00:59.009 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836496 2024-02-05T06:00:59.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:00:59.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836496 2024-02-05T06:00:59.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:00.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:01:00.275 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:00.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-02-05T06:01:00.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:01:00.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:00.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:01:00.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:01:01.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-02-05T06:01:01.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:01:01.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:01.151 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-02-05T06:01:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:01:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:01:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 5 = 1 ']' 2024-02-05T06:01:01.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 5 = 6 ']' 2024-02-05T06:01:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:01.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-6 1 1 2024-02-05T06:01:01.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-6 1 1 2024-02-05T06:01:01.882 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-6' created 2024-02-05T06:01:01.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:01:02.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:01:02.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:01:02.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:01:02.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:01:02.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:01:02.897 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:01:02.897 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:01:02.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:01:02.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:01:02.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:01:02.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:01:02.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:01:02.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:01:02.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:01:02.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:01:02.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:01:03.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:01:03.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:01:03.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:01:03.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:01:03.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-02-05T06:01:03.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-02-05T06:01:03.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-02-05T06:01:03.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:01:03.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-02-05T06:01:03.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:01:03.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:01:03.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-02-05T06:01:03.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:01:03.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-02-05T06:01:03.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-02-05T06:01:03.377 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836498 2024-02-05T06:01:03.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:03.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-02-05T06:01:03.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:04.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:01:04.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:04.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836498 2024-02-05T06:01:04.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:05.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:01:05.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:06.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-02-05T06:01:06.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:01:06.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:06.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:06.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-02-05T06:01:06.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:01:06.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:01:06.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:01:06.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:01:06.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:01:06.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:01:06.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:01:06.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-02-05T06:01:06.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:01:06.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:06.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:07.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-02-05T06:01:07.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:01:07.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:01:07.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 6 = 1 ']' 2024-02-05T06:01:07.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:07.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 6 = 6 ']' 2024-02-05T06:01:07.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5868: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2024-02-05T06:01:07.602 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 6 deep_scrub_interval to 1209600 2024-02-05T06:01:07.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:07.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-7 1 1 2024-02-05T06:01:07.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-7 1 1 2024-02-05T06:01:07.979 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-7' created 2024-02-05T06:01:07.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:01:08.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:01:08.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:01:08.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:01:08.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:01:08.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:01:08.993 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:01:08.993 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:01:08.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:01:08.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:01:08.993 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:01:09.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:01:09.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:01:09.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:01:09.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:01:09.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:01:09.094 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:01:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:01:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:01:09.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:01:09.365 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:01:09.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-02-05T06:01:09.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-02-05T06:01:09.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-02-05T06:01:09.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:01:09.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-02-05T06:01:09.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:01:09.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:01:09.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-02-05T06:01:09.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:01:09.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-02-05T06:01:09.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-02-05T06:01:09.474 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836500 2024-02-05T06:01:09.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:09.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-02-05T06:01:09.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:10.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:01:10.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:11.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-02-05T06:01:11.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:12.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:01:12.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:12.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-02-05T06:01:12.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:01:12.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:12.303 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:12.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 7 == 0 2024-02-05T06:01:12.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:01:12.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:01:12.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:01:12.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:01:12.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:01:12.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:01:12.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:01:12.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-02-05T06:01:12.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:01:12.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:12.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:13.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 7 2024-02-05T06:01:13.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:01:13.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:01:13.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 7 = 1 ']' 2024-02-05T06:01:13.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:13.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 7 = 6 ']' 2024-02-05T06:01:13.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:13.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-8 1 1 2024-02-05T06:01:13.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-8 1 1 2024-02-05T06:01:13.665 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-8' created 2024-02-05T06:01:13.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:01:14.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:01:14.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:01:14.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:01:14.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:01:14.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:01:14.680 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:01:14.680 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:01:14.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:01:14.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:01:14.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:01:14.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:01:14.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:01:14.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:01:14.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:01:14.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:01:14.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:01:15.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:01:15.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:01:15.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:01:15.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:01:15.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-02-05T06:01:15.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-02-05T06:01:15.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-02-05T06:01:15.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:01:15.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-02-05T06:01:15.147 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:01:15.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:01:15.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-02-05T06:01:15.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:01:15.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-02-05T06:01:15.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-02-05T06:01:15.150 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836502 2024-02-05T06:01:15.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:15.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-02-05T06:01:15.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:01:15.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:15.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:01:15.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:01:16.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-02-05T06:01:16.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:01:16.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:16.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:16.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-02-05T06:01:16.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:01:16.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:01:16.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 8 = 1 ']' 2024-02-05T06:01:16.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:16.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 8 = 6 ']' 2024-02-05T06:01:16.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:16.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-9 1 1 2024-02-05T06:01:16.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-9 1 1 2024-02-05T06:01:16.732 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-9' created 2024-02-05T06:01:16.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:01:17.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:01:17.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:01:17.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:01:17.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:01:17.746 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:01:17.746 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:01:17.746 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:01:17.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:01:17.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:01:17.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:01:17.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:01:17.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:01:17.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:01:17.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:01:17.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:01:17.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:01:18.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:01:18.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:01:18.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:01:18.116 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:01:18.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-02-05T06:01:18.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-02-05T06:01:18.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-02-05T06:01:18.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:01:18.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-02-05T06:01:18.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:01:18.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:01:18.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-02-05T06:01:18.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:01:18.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-02-05T06:01:18.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-02-05T06:01:18.221 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836504 2024-02-05T06:01:18.221 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:18.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-02-05T06:01:18.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:19.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:01:19.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:19.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-02-05T06:01:19.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:01:19.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:19.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:20.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-02-05T06:01:20.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:01:20.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:01:20.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:01:20.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:01:20.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:01:20.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:01:20.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:01:20.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-02-05T06:01:20.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:01:20.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:20.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:20.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-02-05T06:01:20.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:01:20.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:01:20.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 9 = 1 ']' 2024-02-05T06:01:20.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:20.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 9 = 6 ']' 2024-02-05T06:01:20.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5858: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:20.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5860: TEST_scrub_warning: create_pool psr_pool-10 1 1 2024-02-05T06:01:20.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool-10 1 1 2024-02-05T06:01:21.059 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'psr_pool-10' created 2024-02-05T06:01:21.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:01:22.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5861: TEST_scrub_warning: wait_for_clean 2024-02-05T06:01:22.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:01:22.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:01:22.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:01:22.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:01:22.073 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:01:22.074 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:01:22.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:01:22.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:01:22.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:01:22.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:01:22.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:01:22.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:01:22.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:01:22.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:01:22.168 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:01:22.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:01:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:01:22.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:01:22.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:01:22.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-02-05T06:01:22.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-02-05T06:01:22.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-02-05T06:01:22.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:01:22.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-02-05T06:01:22.537 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:01:22.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:01:22.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-02-05T06:01:22.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:01:22.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-02-05T06:01:22.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-02-05T06:01:22.540 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836505 2024-02-05T06:01:22.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:22.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-02-05T06:01:22.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:23.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:01:23.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:24.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-02-05T06:01:24.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:01:24.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:24.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:01:24.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:01:24.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-02-05T06:01:24.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:01:24.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:01:24.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:01:25.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-02-05T06:01:25.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:01:25.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:01:25.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5862: TEST_scrub_warning: '[' 10 = 1 ']' 2024-02-05T06:01:25.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:25.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5866: TEST_scrub_warning: '[' 10 = 6 ']' 2024-02-05T06:01:25.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5873: TEST_scrub_warning: local primary=0 2024-02-05T06:01:25.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5875: TEST_scrub_warning: ceph osd set noscrub 2024-02-05T06:01:25.427 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:01:25.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5876: TEST_scrub_warning: ceph osd set nodeep-scrub 2024-02-05T06:01:25.854 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:01:25.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5877: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2024-02-05T06:01:26.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5878: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2024-02-05T06:01:26.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5879: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2024-02-05T06:01:26.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5880: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2024-02-05T06:01:27.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: seq 1 5 2024-02-05T06:01:27.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:01:27.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 1 = 1 ']' 2024-02-05T06:01:27.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5887: TEST_scrub_warning: overdue_seconds=1900800 2024-02-05T06:01:27.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 1900800 + 100 2024-02-05T06:01:27.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2024-02-05T06:01:27.280 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:27.280 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T06:01:27.280 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:27.280 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-14T05:58:07.303840+0000" 2024-02-05T06:01:27.280 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:27.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:01:27.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 2 = 1 ']' 2024-02-05T06:01:27.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:27.289 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 200 2024-02-05T06:01:27.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2024-02-05T06:01:27.430 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:27.430 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T06:01:27.430 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:27.430 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:56:27.453893+0000" 2024-02-05T06:01:27.430 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:27.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:01:27.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 3 = 1 ']' 2024-02-05T06:01:27.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:27.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 300 2024-02-05T06:01:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2024-02-05T06:01:27.580 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:27.580 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T06:01:27.580 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:27.580 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:54:47.604501+0000" 2024-02-05T06:01:27.581 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:27.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:01:27.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 4 = 1 ']' 2024-02-05T06:01:27.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:27.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 400 2024-02-05T06:01:27.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2024-02-05T06:01:27.687 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:27.687 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T06:01:27.687 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:27.687 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:53:07.711106+0000" 2024-02-05T06:01:27.687 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:27.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5883: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2024-02-05T06:01:27.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5885: TEST_scrub_warning: '[' 5 = 1 ']' 2024-02-05T06:01:27.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5889: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:27.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: expr 993600 + 500 2024-02-05T06:01:27.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2024-02-05T06:01:27.795 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:27.795 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T06:01:27.795 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:27.795 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:51:27.818888+0000" 2024-02-05T06:01:27.795 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:27.804 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:27.805 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: expr 5 + 5 2024-02-05T06:01:27.806 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: seq 6 10 2024-02-05T06:01:27.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:27.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:27.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 6 = 6 ']' 2024-02-05T06:01:27.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5898: TEST_scrub_warning: overdue_seconds=1900800 2024-02-05T06:01:27.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 1900800 + 600 2024-02-05T06:01:27.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2024-02-05T06:01:27.908 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:27.908 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:01:27.908 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:27.908 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-14T05:49:47.932245+0000" 2024-02-05T06:01:27.908 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:27.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:27.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:27.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 7 = 6 ']' 2024-02-05T06:01:27.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:27.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 700 2024-02-05T06:01:27.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2024-02-05T06:01:28.017 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:28.018 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:01:28.018 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:28.018 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:48:08.041162+0000" 2024-02-05T06:01:28.018 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:28.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:28.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:28.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 8 = 6 ']' 2024-02-05T06:01:28.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:28.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 800 2024-02-05T06:01:28.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2024-02-05T06:01:28.126 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:28.126 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:01:28.126 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:28.126 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:46:28.150307+0000" 2024-02-05T06:01:28.126 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:28.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:28.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 9 = 6 ']' 2024-02-05T06:01:28.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:28.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 900 2024-02-05T06:01:28.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2024-02-05T06:01:28.235 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:28.235 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:01:28.235 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:28.235 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:44:48.259300+0000" 2024-02-05T06:01:28.235 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:28.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5894: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2024-02-05T06:01:28.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: expr 5 + 1 2024-02-05T06:01:28.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5896: TEST_scrub_warning: '[' 10 = 6 ']' 2024-02-05T06:01:28.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5900: TEST_scrub_warning: overdue_seconds=993600 2024-02-05T06:01:28.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: expr 993600 + 1000 2024-02-05T06:01:28.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5902: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2024-02-05T06:01:28.345 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:01:28.345 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:01:28.345 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:01:28.345 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-24T17:43:08.368885+0000" 2024-02-05T06:01:28.346 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:01:28.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5904: TEST_scrub_warning: flush_pg_stats 2024-02-05T06:01:28.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:01:28.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:01:28.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:01:28.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:01:28.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:01:28.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:01:28.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836508 2024-02-05T06:01:28.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836508 2024-02-05T06:01:28.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836508' 2024-02-05T06:01:28.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:01:28.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836508 2024-02-05T06:01:28.721 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:01:28.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:01:28.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836508 2024-02-05T06:01:28.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:01:28.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836508 2024-02-05T06:01:28.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-02-05T06:01:28.724 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836508 2024-02-05T06:01:28.724 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:28.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836508 2024-02-05T06:01:28.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:01:29.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:01:29.990 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:01:30.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836508 2024-02-05T06:01:30.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5906: TEST_scrub_warning: ceph health 2024-02-05T06:01:30.557 INFO:tasks.workunit.client.0.smithi188.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-02-05T06:01:30.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5907: TEST_scrub_warning: ceph health detail 2024-02-05T06:01:30.887 INFO:tasks.workunit.client.0.smithi188.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 10 pgs not scrubbed in time; 10 pool(s) do not have an application enabled; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2024-02-05T06:01:30.887 INFO:tasks.workunit.client.0.smithi188.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T06:01:30.887 INFO:tasks.workunit.client.0.smithi188.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2024-02-05T06:01:30.887 INFO:tasks.workunit.client.0.smithi188.stdout: pg 10.0 not deep-scrubbed since 2024-01-24T17:43:08.368885+0000 2024-02-05T06:01:30.887 INFO:tasks.workunit.client.0.smithi188.stdout: pg 9.0 not deep-scrubbed since 2024-01-24T17:44:48.259300+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 8.0 not deep-scrubbed since 2024-01-24T17:46:28.150307+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 7.0 not deep-scrubbed since 2024-01-24T17:48:08.041162+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 6.0 not deep-scrubbed since 2024-01-14T05:49:47.932245+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout:[WRN] PG_NOT_SCRUBBED: 10 pgs not scrubbed in time 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 10.0 not scrubbed since 2024-01-24T17:43:08.368885+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 9.0 not scrubbed since 2024-01-24T17:44:48.259300+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 8.0 not scrubbed since 2024-01-24T17:46:28.150307+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 7.0 not scrubbed since 2024-01-24T17:48:08.041162+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 6.0 not scrubbed since 2024-01-14T05:49:47.932245+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 5.0 not scrubbed since 2024-01-24T17:51:27.818888+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 4.0 not scrubbed since 2024-01-24T17:53:07.711106+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 3.0 not scrubbed since 2024-01-24T17:54:47.604501+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 2.0 not scrubbed since 2024-01-24T17:56:27.453893+0000 2024-02-05T06:01:30.888 INFO:tasks.workunit.client.0.smithi188.stdout: pg 1.0 not scrubbed since 2024-01-14T05:58:07.303840+0000 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout:[WRN] POOL_APP_NOT_ENABLED: 10 pool(s) do not have an application enabled 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-1' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-2' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-3' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-4' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-5' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-6' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-7' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-8' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-9' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: application not enabled on pool 'psr_pool-10' 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-1' has no replicas configured 2024-02-05T06:01:30.889 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-2' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-3' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-4' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-5' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-6' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-7' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-8' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-9' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout: pool 'psr_pool-10' has no replicas configured 2024-02-05T06:01:30.890 INFO:tasks.workunit.client.0.smithi188.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2024-02-05T06:01:30.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: ceph health 2024-02-05T06:01:30.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2024-02-05T06:01:31.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: ceph health 2024-02-05T06:01:31.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5909: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2024-02-05T06:01:31.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: ceph health detail 2024-02-05T06:01:31.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: grep 'not scrubbed since' 2024-02-05T06:01:31.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: wc -l 2024-02-05T06:01:31.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5914: TEST_scrub_warning: COUNT=10 2024-02-05T06:01:31.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_scrub_warning: expr 5+5 2024-02-05T06:01:31.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_scrub_warning: (( 10 != 5 && 10 != 5+5 )) 2024-02-05T06:01:31.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: ceph health detail 2024-02-05T06:01:31.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: grep 'not deep-scrubbed since' 2024-02-05T06:01:31.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: wc -l 2024-02-05T06:01:32.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5920: TEST_scrub_warning: COUNT=5 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5921: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:01:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:01:32.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:01:32.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:01:32.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:01:32.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:01:32.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:01:32.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:01:32.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:01:32.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:01:32.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:01:32.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:01:32.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:01:32.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:01:32.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:01:32.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:01:32.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:01:32.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:32.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:32.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:01:32.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:01:32.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:01:32.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:01:32.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:01:32.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:01:32.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:01:32.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:01:32.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:01:32.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:01:32.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:01:32.315 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:01:32.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:01:32.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:01:32.316 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:01:32.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:01:32.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:01:32.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:01:32.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:01:32.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:01:32.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:01:32.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:01:32.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:01:32.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:32.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:32.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:01:32.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:01:32.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:01:32.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:01:32.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:01:32.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:32.324 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:32.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:136: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:137: TEST_skip_non_repair_during_recovery: local poolname=rbd 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_skip_non_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:01:32.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:01:32.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:01:32.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:01:32.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:01:32.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-02-05T06:01:32.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:01:32.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:01:32.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:01:32.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:01:32.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:32.692 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:32.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:01:32.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:01:32.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:01:32.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:01:32.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:01:32.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:01:32.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:01:32.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:01:32.870 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:01:32.870 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:01:32.870 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:01:32.871 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:01:32.871 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:32.871 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:32.871 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:01:32.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:01:32.871 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T06:01:32.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:32.952 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:32.953 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:01:32.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:01:32.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T06:01:33.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:140: TEST_skip_non_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-02-05T06:01:33.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:01:33.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:01:33.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:01:33.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:01:33.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:01:33.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:01:33.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:01:33.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:01:33.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:01:33.846 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:01:33.846 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:33.846 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:33.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:01:33.846 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:01:33.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:01:33.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:01:33.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:01:33.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:01:33.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:01:33.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:01:33.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:01:33.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T06:01:33.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:01:33.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:01:33.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89b5a2b9-3c36-4eda-a061-0045b21ed0a8 2024-02-05T06:01:33.880 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 89b5a2b9-3c36-4eda-a061-0045b21ed0a8 2024-02-05T06:01:33.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 89b5a2b9-3c36-4eda-a061-0045b21ed0a8' 2024-02-05T06:01:33.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:01:33.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9ecBl19XDNhAAQkbh/aMi619mSyf8dnRCEQ== 2024-02-05T06:01:33.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9ecBl19XDNhAAQkbh/aMi619mSyf8dnRCEQ=="}' 2024-02-05T06:01:33.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89b5a2b9-3c36-4eda-a061-0045b21ed0a8 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:01:34.093 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:01:34.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:01:34.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQC9ecBl19XDNhAAQkbh/aMi619mSyf8dnRCEQ== --osd-uuid 89b5a2b9-3c36-4eda-a061-0045b21ed0a8 2024-02-05T06:01:34.134 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:34.157+0000 7f141f0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:34.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:34.160+0000 7f141f0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:34.138 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:34.161+0000 7f141f0164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:34.138 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:34.161+0000 7f141f0164c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:01:34.138 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:34.161+0000 7f141f0164c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:01:34.138 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:34.161+0000 7f141f0164c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:01:34.211 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:34.234+0000 7f141f0164c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:01:39.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:01:39.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:01:39.730 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:01:39.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:01:39.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:01:40.077 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:01:40.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:01:40.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:01:40.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:01:40.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:01:40.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:01:40.105 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:40.128+0000 7fbe5e5904c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:40.107 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:40.130+0000 7fbe5e5904c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:40.108 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:40.131+0000 7fbe5e5904c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:01:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:01:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:01:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:01:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:01:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:01:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:40.349 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:01:40.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:01:40.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:40.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:40.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:40.627 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:40.651+0000 7fbe5e5904c0 -1 Falling back to public interface 2024-02-05T06:01:41.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:41.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:41.619 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:01:41.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:01:41.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:41.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:41.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:42.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:42.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:42.895 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:01:42.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:01:42.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:42.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:43.006 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:01:43.029+0000 7fbe5e5904c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:01:43.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:44.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:44.197 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:44.198 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:01:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:01:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:44.198 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:44.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:45.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:45.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:45.476 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:01:45.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:01:45.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:45.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:45.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:46.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:46.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:46.852 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:01:46.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:01:46.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:46.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:47.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:48.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:48.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:48.131 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:01:48.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:01:48.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:48.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:48.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:49.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:49.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:49.409 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:01:49.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:01:49.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:49.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:49.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:50.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:50.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:50.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:01:50.688 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:01:50.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:50.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:51.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:52.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:52.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:52.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:01:52.022 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:01:52.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:52.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:52.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:53.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:53.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:53.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:01:53.302 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:01:53.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:53.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:53.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:54.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:54.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:54.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:01:54.582 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:01:54.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:54.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:55.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:56.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:56.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:56.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:01:56.159 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:01:56.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:56.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:56.601 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:57.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:01:57.604 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:01:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:57.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:01:58.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:01:59.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:01:59.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:01:59.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:01:59.390 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:01:59.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:01:59.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:00.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:01.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:01.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:01.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:02:01.363 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:02:01.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:01.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:01.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:02.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:02.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:02.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:02:02.864 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:02:02.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:02.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:03.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:04.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:04.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:04.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:02:04.188 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:02:04.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:04.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:04.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:05.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:05.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:05.928 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:02:05.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:02:05.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:05.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:06.245 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2984674538,v1:127.0.0.1:6803/2984674538] [v2:127.0.0.1:6804/2984674538,v1:127.0.0.1:6805/2984674538] exists,up 89b5a2b9-3c36-4eda-a061-0045b21ed0a8 2024-02-05T06:02:06.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:02:06.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:02:06.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:02:06.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:144: TEST_skip_non_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:02:06.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:02:06.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:02:06.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:02:06.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:02:06.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:02:06.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:02:06.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:02:06.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-02-05T06:02:06.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:02:06.255 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:02:06.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=94638934-4cd9-445c-9802-ca7d73439e02 2024-02-05T06:02:06.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 94638934-4cd9-445c-9802-ca7d73439e02' 2024-02-05T06:02:06.256 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 94638934-4cd9-445c-9802-ca7d73439e02 2024-02-05T06:02:06.256 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:02:06.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeecBlsllQERAAovfg0RWy+MyY00owQnqFVg== 2024-02-05T06:02:06.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeecBlsllQERAAovfg0RWy+MyY00owQnqFVg=="}' 2024-02-05T06:02:06.267 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 94638934-4cd9-445c-9802-ca7d73439e02 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:02:06.624 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:02:06.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:02:06.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQDeecBlsllQERAAovfg0RWy+MyY00owQnqFVg== --osd-uuid 94638934-4cd9-445c-9802-ca7d73439e02 2024-02-05T06:02:06.662 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:06.685+0000 7fdea941a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:06.697 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:06.720+0000 7fdea941a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:06.697 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:06.721+0000 7fdea941a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:06.698 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:06.722+0000 7fdea941a4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:02:06.698 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:06.722+0000 7fdea941a4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:02:06.698 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:06.722+0000 7fdea941a4c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:02:06.759 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:06.783+0000 7fdea941a4c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:02:12.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:02:12.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:02:12.493 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:02:12.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:02:12.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:02:12.837 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:02:12.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:02:12.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:02:12.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-02-05T06:02:12.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:02:12.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:02:12.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:12.889+0000 7fa2a841f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:12.867 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:12.891+0000 7fa2a841f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:12.868 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:12.892+0000 7fa2a841f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:13.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:02:13.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:02:13.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:02:13.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:02:13.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:02:13.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:13.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:02:13.105 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:02:13.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:13.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:13.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:13.391 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:13.415+0000 7fa2a841f4c0 -1 Falling back to public interface 2024-02-05T06:02:14.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:14.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:14.370 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:02:14.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:02:14.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:14.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:14.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:15.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:15.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:15.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:02:15.645 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:02:15.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:15.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:15.712 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:15.736+0000 7fa2a841f4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:02:15.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:16.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:16.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:16.931 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:02:16.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:02:16.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:16.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:17.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:18.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:18.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:18.208 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:02:18.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:02:18.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:18.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:18.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:19.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:19.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:19.576 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:02:19.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:02:19.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:19.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:19.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:20.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:20.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:20.871 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:02:20.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:02:20.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:20.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:21.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:22.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:22.151 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:02:22.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:02:22.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:22.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:22.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:23.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:23.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:23.474 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:02:23.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:02:23.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:23.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:23.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:24.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:24.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:24.753 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:02:24.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:02:24.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:24.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:25.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:26.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:26.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:26.030 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:02:26.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:02:26.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:26.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:26.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:27.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:27.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:27.392 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:02:27.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:02:27.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:27.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:27.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:28.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:28.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:28.770 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:02:28.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:02:28.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:28.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:29.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:30.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:30.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:30.351 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:02:30.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:02:30.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:30.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:31.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:32.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:32.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:32.119 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:02:32.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:02:32.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:32.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:33.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:34.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:34.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:34.005 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:02:34.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:02:34.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:34.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:34.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:35.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:35.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:35.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:02:35.412 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:02:35.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:35.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:36.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:37.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:37.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:37.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:02:37.085 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:02:37.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:37.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:37.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:38.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:38.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:38.384 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:02:38.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:02:38.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:38.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:38.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:39.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:39.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:39.722 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:02:39.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:02:39.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:39.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:02:40.034 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3408852582,v1:127.0.0.1:6811/3408852582] [v2:127.0.0.1:6812/3408852582,v1:127.0.0.1:6813/3408852582] exists,up 94638934-4cd9-445c-9802-ca7d73439e02 2024-02-05T06:02:40.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:02:40.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:02:40.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:02:40.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_skip_non_repair_during_recovery: create_rbd_pool 2024-02-05T06:02:40.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-02-05T06:02:40.305 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' does not exist 2024-02-05T06:02:40.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-02-05T06:02:40.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-02-05T06:02:40.682 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'rbd' created 2024-02-05T06:02:40.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:02:41.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-02-05T06:02:43.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_skip_non_repair_during_recovery: wait_for_clean 2024-02-05T06:02:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:02:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:02:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:02:43.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:02:43.396 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:02:43.396 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:02:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:02:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:02:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:02:43.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:02:43.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:02:43.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:02:43.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:02:43.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:02:43.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:02:43.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:02:43.761 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T06:02:43.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:02:43.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:43.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:02:43.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-02-05T06:02:43.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-02-05T06:02:43.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-02-05T06:02:43.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:02:43.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:02:43.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-02-05T06:02:43.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-02-05T06:02:43.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672967' 2024-02-05T06:02:43.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:43.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-02-05T06:02:43.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:43.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:02:43.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-02-05T06:02:43.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:43.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-02-05T06:02:43.983 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836494 2024-02-05T06:02:43.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-02-05T06:02:43.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:44.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-02-05T06:02:44.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:02:45.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:02:45.284 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:45.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836494 2024-02-05T06:02:45.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:02:46.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:02:46.560 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:02:46.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836494 2024-02-05T06:02:46.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:02:46.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-02-05T06:02:46.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:02:46.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:02:46.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-02-05T06:02:46.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:02:46.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-02-05T06:02:46.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-02-05T06:02:46.836 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672967 2024-02-05T06:02:46.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:02:47.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-02-05T06:02:47.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:02:47.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:47.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:02:47.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:02:47.439 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:02:47.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:02:47.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:02:47.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:02:47.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:02:47.440 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:02:47.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:02:47.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:02:47.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:02:47.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:02:48.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:02:48.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:02:48.514 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:02:48.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:02:49.042 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:02:49.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:02:49.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:02:49.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T06:02:49.355 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2024-02-05T06:02:49.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-02-05T06:02:49.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-02-05T06:02:49.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-02-05T06:02:49.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-02-05T06:02:49.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T06:02:49.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T06:02:49.356 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:02:49.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:02:49.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-02-05T06:02:49.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:02:50.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2024-02-05T06:02:50.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:155: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2024-02-05T06:02:50.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:156: scrub_and_not_schedule: local poolname=rbd 2024-02-05T06:02:50.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:157: scrub_and_not_schedule: local osd=0 2024-02-05T06:02:50.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: get_pg rbd SOMETHING 2024-02-05T06:02:50.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-02-05T06:02:50.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T06:02:50.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-02-05T06:02:50.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:02:50.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:162: scrub_and_not_schedule: local pg=1.3 2024-02-05T06:02:50.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:02:50.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:02:50.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:02:50.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:02:50.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:02:50.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:163: scrub_and_not_schedule: local last_scrub=2024-02-05T06:02:40.652951+0000 2024-02-05T06:02:50.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:164: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2024-02-05T06:02:50.691 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:02:50.691 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T06:02:50.691 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:02:50.691 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T06:01:10.715938+0000" 2024-02-05T06:02:50.691 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:02:50.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i=0 )) 2024-02-05T06:02:50.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:02:50.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:02:50.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:02:50.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:02:50.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:02:50.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:02:50.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-02-05T06:02:40.652951+0000 '>' 2024-02-05T06:02:40.652951+0000 2024-02-05T06:02:50.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-02-05T06:02:51.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-02-05T06:02:51.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:02:51.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:02:51.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:02:51.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:02:51.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:02:51.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:02:52.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-02-05T06:02:40.652951+0000 '>' 2024-02-05T06:02:40.652951+0000 2024-02-05T06:02:52.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-02-05T06:02:53.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-02-05T06:02:53.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:02:53.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2024-02-05T06:02:53.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-02-05T06:02:53.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:02:53.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:02:53.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-02-05T06:02:53.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: scrub_and_not_schedule: test 2024-02-05T06:02:40.652951+0000 '>' 2024-02-05T06:02:40.652951+0000 2024-02-05T06:02:53.513 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:173: scrub_and_not_schedule: sleep 1 2024-02-05T06:02:54.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i++ )) 2024-02-05T06:02:54.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: scrub_and_not_schedule: (( i < 3 )) 2024-02-05T06:02:54.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:179: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T06:02:54.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:02:54.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:02:54.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:02:54.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:02:54.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T06:02:54.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:02:54.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:02:54.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:02:54.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:02:54.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:02:54.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:02:54.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:02:54.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:02:54.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:02:54.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:02:54.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-02-05T06:02:56.196 INFO:tasks.workunit.client.0.smithi188.stdout:_ 2024-02-05T06:02:56.197 INFO:tasks.workunit.client.0.smithi188.stdout:snapset 2024-02-05T06:02:56.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:02:56.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:02:56.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:02:56.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:02:56.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:02:56.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:02:56.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:02:56.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:02:56.472 INFO:tasks.workunit.client.0.smithi188.stderr:start osd.0 2024-02-05T06:02:56.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:02:56.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:02:56.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:02:56.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:02:56.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:02:56.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:02:56.499 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:56.523+0000 7f535961d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:56.501 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:56.525+0000 7f535961d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:56.503 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:56.526+0000 7f535961d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:02:56.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:02:56.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:02:56.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:02:56.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:02:56.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:02:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:02:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T06:02:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:56.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:57.020 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:57.044+0000 7f535961d4c0 -1 Falling back to public interface 2024-02-05T06:02:57.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:58.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:58.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:58.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:02:58.028 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:02:58.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:58.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:58.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:02:59.285 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:02:59.309+0000 7f535961d4c0 -1 osd.0 22 log_to_monitors true 2024-02-05T06:02:59.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:02:59.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:02:59.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:02:59.305 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T06:02:59.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:02:59.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:02:59.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:00.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:03:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:3 2024-02-05T06:03:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:00.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:00.952 INFO:tasks.workunit.client.0.smithi188.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/2580554194,v1:127.0.0.1:6803/2580554194] [v2:127.0.0.1:6804/2580554194,v1:127.0.0.1:6805/2580554194] exists,up 89b5a2b9-3c36-4eda-a061-0045b21ed0a8 2024-02-05T06:03:00.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:03:00.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:03:00.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:03:00.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:03:00.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:03:00.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:03:00.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:03:00.953 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:03:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:03:00.954 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:03:00.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:03:00.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:03:00.954 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:03:01.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:03:01.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:03:01.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:03:01.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:03:01.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:03:01.044 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:03:01.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:03:01.310 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T06:03:01.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:03:01.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:01.310 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:03:01.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-02-05T06:03:01.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-02-05T06:03:01.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-02-05T06:03:01.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:03:01.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:03:01.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T06:03:01.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T06:03:01.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672972' 2024-02-05T06:03:01.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:01.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-02-05T06:03:01.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:01.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:03:01.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-02-05T06:03:01.518 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-02-05T06:03:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-02-05T06:03:01.519 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.0 seq 111669149698 2024-02-05T06:03:01.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:01.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-02-05T06:03:01.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:03:02.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:03:02.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:03:03.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-02-05T06:03:03.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:03:03.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T06:03:03.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:03:03.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:03:03.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T06:03:03.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:03:03.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T06:03:03.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T06:03:03.063 INFO:tasks.workunit.client.0.smithi188.stderr:waiting osd.1 seq 42949672972 2024-02-05T06:03:03.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:03:03.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T06:03:03.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:03:03.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:03.330 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:03.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-02-05T06:03:03.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:03:03.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:03:03.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:03:03.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:03:03.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:03:03.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:03:03.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:03:03.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-02-05T06:03:03.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:03:03.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:03:03.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:03:04.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-02-05T06:03:04.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:03:04.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:03:04.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-02-05T06:03:04.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-02-05T06:03:04.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:03:04.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:03:04.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:03:04.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:03:04.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:03:04.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:03:04.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:03:04.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:03:04.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:03:04.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:03:04.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:03:04.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:03:04.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:03:04.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:03:04.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:03:04.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:03:04.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:03:04.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:04.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:04.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:03:04.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:03:04.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:03:04.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:03:04.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:03:04.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:03:04.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:03:04.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:03:04.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:03:04.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:03:04.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:03:04.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:03:04.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:03:04.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:03:04.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:03:04.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:03:04.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:03:04.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:03:04.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:03:04.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:03:04.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:03:04.452 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:03:04.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:03:04.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:03:04.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:03:04.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:03:04.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:03:04.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:03:04.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:03:04.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:03:04.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:04.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:04.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:03:04.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:03:04.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:03:04.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:03:04.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:03:04.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:04.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:04.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T06:03:04.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:03:04.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:03:04.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:03:04.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:903: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=false 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:03:04.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:03:04.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:03:05.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:03:05.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:03:05.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:03:05.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:03:05.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:03:05.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:03:05.026 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:03:05.026 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:03:05.026 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:03:05.026 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:03:05.026 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:05.026 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:05.027 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:03:05.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:03:05.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T06:03:05.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:03:05.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:03:05.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:03:05.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:03:05.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:03:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:03:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:03:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:03:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:03:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:05.108 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:03:05.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:03:05.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T06:03:05.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T06:03:05.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:03:05.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:03:05.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:03:05.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:03:05.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:03:05.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:03:05.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:03:05.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:05.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:05.996 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:05.996 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:05.996 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:05.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:03:05.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:03:05.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:03:06.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:03:06.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:03:06.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:03:06.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:03:06.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74ae8727-9d4d-490e-832c-755a695010d4 2024-02-05T06:03:06.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 74ae8727-9d4d-490e-832c-755a695010d4' 2024-02-05T06:03:06.031 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 74ae8727-9d4d-490e-832c-755a695010d4 2024-02-05T06:03:06.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:03:06.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaesBlX+k1BBAA6Wf3fdTiGnmy5+/R7EejBw== 2024-02-05T06:03:06.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaesBlX+k1BBAA6Wf3fdTiGnmy5+/R7EejBw=="}' 2024-02-05T06:03:06.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74ae8727-9d4d-490e-832c-755a695010d4 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:03:06.251 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:03:06.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:03:06.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaesBlX+k1BBAA6Wf3fdTiGnmy5+/R7EejBw== --osd-uuid 74ae8727-9d4d-490e-832c-755a695010d4 2024-02-05T06:03:06.290 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:06.314+0000 7f556da104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:06.301 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:06.326+0000 7f556da104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:06.302 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:06.327+0000 7f556da104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:06.327+0000 7f556da104c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:03:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:06.327+0000 7f556da104c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:03:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:06.327+0000 7f556da104c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:03:06.394 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:06.418+0000 7f556da104c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:03:11.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:03:11.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:03:11.784 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:03:11.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:03:11.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:03:12.162 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:03:12.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:03:12.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:03:12.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:03:12.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:03:12.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:03:12.188 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:12.212+0000 7f7b758be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:12.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:12.214+0000 7f7b758be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:12.191 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:12.215+0000 7f7b758be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:12.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:03:12.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:03:12.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:03:12.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:03:12.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:03:12.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:12.429 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:03:12.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:03:12.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:12.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:12.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:12.722 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:12.746+0000 7f7b758be4c0 -1 Falling back to public interface 2024-02-05T06:03:13.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:13.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:13.696 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:03:13.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:03:13.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:13.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:13.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:14.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:03:14.971 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:03:14.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:14.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:15.066 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:15.090+0000 7f7b758be4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:03:15.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:16.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:16.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:16.270 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:03:16.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:03:16.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:16.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:16.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:17.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:17.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:17.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:03:17.623 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:03:17.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:17.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:18.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:19.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:19.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:19.003 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:03:19.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:03:19.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:19.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:19.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:20.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:20.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:20.283 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:03:20.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:03:20.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:20.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:20.602 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:21.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:21.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:21.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:03:21.604 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:03:21.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:21.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:21.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:22.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:22.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:22.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:03:22.961 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:03:22.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:22.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:23.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:24.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:24.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:24.239 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:03:24.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:03:24.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:24.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:24.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:25.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:25.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:25.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:03:25.521 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:03:25.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:25.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:25.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:26.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:26.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:26.817 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:03:26.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:03:26.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:26.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:27.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:28.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:28.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:28.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:03:28.416 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:03:28.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:28.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:29.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:30.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:30.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:30.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:03:30.055 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:03:30.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:30.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:30.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:31.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:31.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:31.841 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:03:31.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:03:31.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:31.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:32.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:33.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:33.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:03:33.243 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:03:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:33.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:33.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:34.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:34.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:34.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:03:34.945 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:03:34.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:34.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:35.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:36.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:03:36.690 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:03:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:36.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:37.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:38.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:38.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:38.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:03:38.112 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:03:38.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:38.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:03:38.438 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1380535474,v1:127.0.0.1:6803/1380535474] [v2:127.0.0.1:6804/1380535474,v1:127.0.0.1:6805/1380535474] exists,up 74ae8727-9d4d-490e-832c-755a695010d4 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:03:38.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:03:38.443 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:03:38.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:03:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:03:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:03:38.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:03:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6567995b-b641-48d3-a7ec-a4dcd40b439d 2024-02-05T06:03:38.445 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6567995b-b641-48d3-a7ec-a4dcd40b439d' 2024-02-05T06:03:38.445 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 6567995b-b641-48d3-a7ec-a4dcd40b439d 2024-02-05T06:03:38.445 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:03:38.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6esBl7KLdHBAA2mTEn63dwiIDuJDeEHq1Cg== 2024-02-05T06:03:38.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6esBl7KLdHBAA2mTEn63dwiIDuJDeEHq1Cg=="}' 2024-02-05T06:03:38.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6567995b-b641-48d3-a7ec-a4dcd40b439d -i td/osd-scrub-repair/1/new.json 2024-02-05T06:03:38.810 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:03:38.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:03:38.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6esBl7KLdHBAA2mTEn63dwiIDuJDeEHq1Cg== --osd-uuid 6567995b-b641-48d3-a7ec-a4dcd40b439d 2024-02-05T06:03:38.850 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:38.875+0000 7f1166c104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:38.886 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:38.910+0000 7f1166c104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:38.887 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:38.911+0000 7f1166c104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:38.887 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:38.912+0000 7f1166c104c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:03:38.887 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:38.912+0000 7f1166c104c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:03:38.888 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:38.912+0000 7f1166c104c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:03:38.961 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:38.985+0000 7f1166c104c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:03:45.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:03:45.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:03:45.012 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:03:45.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:03:45.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:03:45.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:03:45.359 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:03:45.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:03:45.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:03:45.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:03:45.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:03:45.384 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:45.409+0000 7f1181cb24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:45.386 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:45.411+0000 7f1181cb24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:45.387 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:45.412+0000 7f1181cb24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:03:45.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:03:45.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:03:45.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:03:45.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:03:45.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:03:45.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:45.627 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:03:45.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:03:45.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:45.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:45.906 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:45.931+0000 7f1181cb24c0 -1 Falling back to public interface 2024-02-05T06:03:45.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:46.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:46.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:46.951 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:03:46.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:03:46.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:46.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:47.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:48.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:48.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:48.221 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:03:48.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:03:48.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:48.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:48.392 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:03:48.416+0000 7f1181cb24c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:03:48.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:49.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:49.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:49.506 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:03:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:03:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:49.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:49.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:50.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:50.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:50.783 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:03:50.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:03:50.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:50.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:51.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:52.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:52.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:52.171 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:03:52.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:03:52.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:52.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:52.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:53.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:53.449 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:03:53.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:53.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:03:53.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:53.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:53.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:54.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:54.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:54.730 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:03:54.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:03:54.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:54.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:55.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:56.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:56.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:56.009 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:03:56.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:03:56.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:56.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:56.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:57.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:57.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:57.286 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:03:57.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:03:57.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:57.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:57.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:58.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:58.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:58.567 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:03:58.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:03:58.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:58.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:03:58.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:03:59.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:03:59.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:03:59.843 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:03:59.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:03:59.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:03:59.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:00.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:01.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:01.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:01.830 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:04:01.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:04:01.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:01.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:02.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:03.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:03.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:03.475 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:04:03.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:04:03.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:03.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:04.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:05.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:05.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:05.035 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:04:05.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:04:05.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:05.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:05.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:06.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:06.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:06.828 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:04:06.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:04:06.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:06.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:07.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:08.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:08.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:08.114 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:04:08.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:04:08.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:08.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:09.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:10.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:10.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:10.186 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:04:10.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:04:10.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:10.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:10.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:11.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:11.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:11.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:04:11.880 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:04:11.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:11.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:04:12.170 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4063520572,v1:127.0.0.1:6811/4063520572] [v2:127.0.0.1:6812/4063520572,v1:127.0.0.1:6813/4063520572] exists,up 6567995b-b641-48d3-a7ec-a4dcd40b439d 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:04:12.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:04:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:04:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:04:12.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:04:12.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:04:12.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:04:12.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29a22b72-7c0d-4831-8828-e2d38da57c07 2024-02-05T06:04:12.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 29a22b72-7c0d-4831-8828-e2d38da57c07' 2024-02-05T06:04:12.178 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 29a22b72-7c0d-4831-8828-e2d38da57c07 2024-02-05T06:04:12.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:04:12.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcesBlIenrDBAAoxFgRltMq/dkmBvuIg/c+w== 2024-02-05T06:04:12.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcesBlIenrDBAAoxFgRltMq/dkmBvuIg/c+w=="}' 2024-02-05T06:04:12.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29a22b72-7c0d-4831-8828-e2d38da57c07 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:04:12.538 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:04:12.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:04:12.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcesBlIenrDBAAoxFgRltMq/dkmBvuIg/c+w== --osd-uuid 29a22b72-7c0d-4831-8828-e2d38da57c07 2024-02-05T06:04:12.578 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:12.603+0000 7f2525d124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:12.623 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:12.648+0000 7f2525d124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:12.624 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:12.649+0000 7f2525d124c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:12.624 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:12.649+0000 7f2525d124c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:04:12.624 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:12.649+0000 7f2525d124c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:04:12.625 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:12.649+0000 7f2525d124c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:04:12.698 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:12.723+0000 7f2525d124c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:04:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:04:18.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:04:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:04:18.813 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T06:04:18.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:04:19.156 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T06:04:19.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:04:19.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:04:19.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:04:19.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:04:19.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:04:19.182 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:19.206+0000 7f492ecfa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:19.183 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:19.208+0000 7f492ecfa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:19.184 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:19.209+0000 7f492ecfa4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:04:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:04:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:04:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:04:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:04:19.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:19.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:04:19.426 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:04:19.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:19.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:19.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:19.710 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:19.735+0000 7f492ecfa4c0 -1 Falling back to public interface 2024-02-05T06:04:20.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:20.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:20.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:04:20.696 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:04:20.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:20.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:20.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:21.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:21.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:21.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:04:21.966 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:04:21.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:21.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:22.097 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:22.122+0000 7f492ecfa4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T06:04:22.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:23.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:23.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:23.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:04:23.333 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:04:23.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:23.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:23.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:24.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:24.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:24.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:04:24.562 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:04:24.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:24.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:24.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:25.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:25.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:25.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:04:25.841 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:04:25.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:25.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:26.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:27.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:27.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:27.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:04:27.205 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:04:27.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:27.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:27.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:28.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:28.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:28.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:04:28.484 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:04:28.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:28.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:28.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:29.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:29.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:29.761 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:04:29.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:04:29.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:29.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:30.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:31.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:31.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:31.042 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:04:31.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:04:31.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:31.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:31.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:32.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:32.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:32.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:04:32.320 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:04:32.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:32.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:32.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:33.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:04:33.669 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:04:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:33.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:34.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:34.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:34.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:04:34.943 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:04:34.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:34.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:35.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:36.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:36.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:36.370 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:04:36.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:04:36.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:36.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:37.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:38.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:38.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:38.348 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:04:38.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:04:38.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:38.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:38.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:39.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:39.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:39.967 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:04:39.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:04:39.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:39.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:40.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:41.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:41.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:41.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:04:41.753 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:04:41.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:41.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:42.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:43.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:43.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:43.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:04:43.576 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:04:43.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:43.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:44.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:45.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:45.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:45.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:04:45.527 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:04:45.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:45.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:04:45.806 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3353148810,v1:127.0.0.1:6819/3353148810] [v2:127.0.0.1:6820/3353148810,v1:127.0.0.1:6821/3353148810] exists,up 29a22b72-7c0d-4831-8828-e2d38da57c07 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:04:45.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:04:45.812 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:04:45.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:04:45.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:04:45.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:04:45.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1de05a9d-1ae6-4d26-ad37-ca2861ec581c 2024-02-05T06:04:45.814 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 1de05a9d-1ae6-4d26-ad37-ca2861ec581c 2024-02-05T06:04:45.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 1de05a9d-1ae6-4d26-ad37-ca2861ec581c' 2024-02-05T06:04:45.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:04:45.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9esBlYIveMhAAd7q7rwXVq0D0oVDv477/Pg== 2024-02-05T06:04:45.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9esBlYIveMhAAd7q7rwXVq0D0oVDv477/Pg=="}' 2024-02-05T06:04:45.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1de05a9d-1ae6-4d26-ad37-ca2861ec581c -i td/osd-scrub-repair/3/new.json 2024-02-05T06:04:46.311 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:04:46.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T06:04:46.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9esBlYIveMhAAd7q7rwXVq0D0oVDv477/Pg== --osd-uuid 1de05a9d-1ae6-4d26-ad37-ca2861ec581c 2024-02-05T06:04:46.351 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:46.375+0000 7f4a495be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:46.396 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:46.420+0000 7f4a495be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:46.396 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:46.421+0000 7f4a495be4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:46.397 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:46.422+0000 7f4a495be4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:04:46.397 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:46.422+0000 7f4a495be4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:04:46.397 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:46.422+0000 7f4a495be4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:04:46.470 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:46.495+0000 7f4a495be4c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T06:04:52.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T06:04:52.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:04:52.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T06:04:52.437 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T06:04:52.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:04:52.785 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T06:04:52.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T06:04:52.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:04:52.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:04:52.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:04:52.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:04:52.811 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:52.836+0000 7f204560c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:52.813 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:52.839+0000 7f204560c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:52.814 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:52.840+0000 7f204560c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:04:53.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T06:04:53.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:04:53.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:04:53.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:04:53.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:04:53.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:53.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:04:53.055 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:04:53.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:53.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:04:53.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:53.351 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:53.376+0000 7f204560c4c0 -1 Falling back to public interface 2024-02-05T06:04:54.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:54.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:54.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:04:54.321 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:04:54.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:54.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:04:54.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:55.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:55.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:55.598 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:04:55.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:04:55.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:55.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:04:55.687 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:04:55.712+0000 7f204560c4c0 -1 osd.3 0 log_to_monitors true 2024-02-05T06:04:55.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:56.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:56.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:56.898 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:04:56.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:04:56.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:56.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:04:57.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:58.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:58.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:58.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:04:58.176 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:04:58.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:58.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:04:58.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:04:59.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:04:59.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:04:59.532 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:04:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:04:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:04:59.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:04:59.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:00.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:00.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:00.814 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:05:00.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:05:00.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:00.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:01.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:02.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:02.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:02.091 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:05:02.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:05:02.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:02.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:02.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:03.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:03.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:03.394 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:05:03.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:05:03.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:03.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:03.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:04.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:04.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:04.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:05:04.705 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:05:04.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:04.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:04.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:05.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:05.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:05.983 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:05:05.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:05:05.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:05.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:06.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:07.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:07.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:07.356 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:05:07.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:05:07.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:07.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:07.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:08.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:08.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:08.696 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:05:08.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:05:08.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:08.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:09.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:10.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:10.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:10.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:05:10.285 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:05:10.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:10.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:11.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:12.021 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:05:12.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:05:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:12.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:12.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:13.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:13.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:13.532 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:05:13.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:05:13.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:13.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:14.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:15.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:15.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:15.095 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:05:15.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:05:15.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:15.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:15.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:16.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:16.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:16.648 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:05:16.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:05:16.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:16.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:17.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:18.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:18.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:18.063 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:05:18.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:05:18.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:18.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:18.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:19.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:19.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:19.379 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:05:19.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:05:19.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:19.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:19.655 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3126945361,v1:127.0.0.1:6827/3126945361] [v2:127.0.0.1:6828/3126945361,v1:127.0.0.1:6829/3126945361] exists,up 1de05a9d-1ae6-4d26-ad37-ca2861ec581c 2024-02-05T06:05:19.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:05:19.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:05:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:05:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2024-02-05T06:05:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T06:05:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:05:19.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-02-05T06:05:19.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:05:19.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T06:05:20.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T06:05:20.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T06:05:20.917 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T06:05:20.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:05:21.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-02-05T06:05:21.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:05:21.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:05:21.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:05:21.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:05:21.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:05:21.932 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:05:21.932 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:05:21.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:05:21.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:05:21.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:05:22.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:05:22.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:05:22.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:05:22.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:05:22.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:05:22.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:05:22.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:05:22.302 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:05:22.302 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T06:05:22.302 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T06:05:22.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:05:22.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:22.302 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:05:22.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-02-05T06:05:22.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-02-05T06:05:22.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-02-05T06:05:22.436 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:22.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:05:22.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-02-05T06:05:22.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-02-05T06:05:22.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672982' 2024-02-05T06:05:22.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:22.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:05:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-02-05T06:05:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-02-05T06:05:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672982 2-64424509452' 2024-02-05T06:05:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:22.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:05:22.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345928 2024-02-05T06:05:22.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345928 2024-02-05T06:05:22.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672982 2-64424509452 3-85899345928' 2024-02-05T06:05:22.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:22.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-02-05T06:05:22.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:22.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:05:22.832 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-02-05T06:05:22.833 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:22.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-02-05T06:05:22.836 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836509 2024-02-05T06:05:22.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-02-05T06:05:22.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:05:23.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-02-05T06:05:23.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:05:24.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:05:24.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:05:24.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836509 2024-02-05T06:05:24.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:24.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-02-05T06:05:24.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:24.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:05:24.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-02-05T06:05:24.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:24.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-02-05T06:05:24.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-02-05T06:05:24.411 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672982 2024-02-05T06:05:24.411 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:24.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672982 -lt 42949672982 2024-02-05T06:05:24.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:24.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-02-05T06:05:24.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:24.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:05:24.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-02-05T06:05:24.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:24.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-02-05T06:05:24.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-02-05T06:05:24.681 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509452 2024-02-05T06:05:24.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:05:24.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-02-05T06:05:24.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:24.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345928 2024-02-05T06:05:24.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:24.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:05:24.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345928 2024-02-05T06:05:24.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:24.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345928 2024-02-05T06:05:24.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2024-02-05T06:05:24.951 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345928 2024-02-05T06:05:24.951 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:05:25.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345928 2024-02-05T06:05:25.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:05:25.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:25.218 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:25.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:05:25.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:25.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:25.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:25.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:25.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:25.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:25.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:25.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:05:25.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:25.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:25.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:05:26.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:05:26.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:05:26.552 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:05:26.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:05:27.063 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:05:27.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:05:27.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:05:27.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T06:05:27.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T06:05:27.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T06:05:27.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T06:05:27.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:05:27.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:05:27.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-02-05T06:05:27.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T06:05:27.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-02-05T06:05:27.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:05:27.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T06:05:27.670 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:05:27.670 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-02-05T06:05:27.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 466787"' 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 466787' 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 466788"' 2024-02-05T06:05:27.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 466788' 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/466790: /' 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 466791"' 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 466791' 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 466787 466788 466791' 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 466787 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/466793: /' 2024-02-05T06:05:27.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:05:27.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/466796: /' 2024-02-05T06:05:30.720 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.720 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:05:30.720 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T06:05:30.720 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:05:30.720 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.721 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: remove 3#1:eb822e21:::SOMETHING:head# 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:05:30.722 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:05:30.724 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:05:30.725 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:05:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:05:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:05:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:05:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:05:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T06:05:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:466796: start osd.3 2024-02-05T06:05:30.726 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:05:30.748 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.749 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: remove 1#1:eb822e21:::SOMETHING:head# 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:05:30.750 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:05:30.751 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:05:30.751 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:05:30.751 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:05:30.751 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:30.752 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:05:30.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:05:30.754 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:05:30.754 INFO:tasks.workunit.client.0.smithi188.stderr:466790: start osd.0 2024-02-05T06:05:30.754 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:05:30.779 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:05:30.780 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T06:05:30.781 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: remove 2#1:eb822e21:::SOMETHING:head# 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:05:30.782 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:05:30.783 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:30.784 INFO:tasks.workunit.client.0.smithi188.stderr:466793: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:05:30.785 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:05:30.786 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:05:30.786 INFO:tasks.workunit.client.0.smithi188.stderr:466793: start osd.2 2024-02-05T06:05:30.786 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 2024-02-05T06:05:30.806+0000 7fe05a4134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 2024-02-05T06:05:30.809+0000 7fe05a4134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.240 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 2024-02-05T06:05:30.811+0000 7fe05a4134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 0 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 2024-02-05T06:05:31.331+0000 7fe05a4134c0 -1 Falling back to public interface 2024-02-05T06:05:35.241 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 1 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 2 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.242 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 2024-02-05T06:05:34.218+0000 7fe05a4134c0 -1 osd.0 28 log_to_monitors true 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 3 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:35.243 INFO:tasks.workunit.client.0.smithi188.stderr:466790: osd.0 up in weight 1 up_from 33 up_thru 0 down_at 29 last_c466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 2024-02-05T06:05:30.778+0000 7f0d512a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 2024-02-05T06:05:30.781+0000 7f0d512a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 2024-02-05T06:05:30.783+0000 7f0d512a14c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:05:35.244 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 0 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 2024-02-05T06:05:31.299+0000 7f0d512a14c0 -1 Falling back to public interface 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 1 2024-02-05T06:05:35.245 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 2 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 2024-02-05T06:05:34.067+0000 7f0d512a14c0 -1 osd.3 28 log_to_monitors true 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.246 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:05:35.247 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 3 2024-02-05T06:05:35.247 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.247 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:05:35.254 INFO:tasks.workunit.client.0.smithi188.stderr:466796: osd.3 up in weight 1 up_from 32 up_thru 0 down_at 29 last_c466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:05:35.254 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:05:35.254 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:05:35.254 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:05:35.254 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:05:35.254 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 2024-02-05T06:05:30.836+0000 7f4b5bac04c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 2024-02-05T06:05:30.839+0000 7f4b5bac04c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 2024-02-05T06:05:30.840+0000 7f4b5bac04c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 0 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 2024-02-05T06:05:31.358+0000 7f4b5bac04c0 -1 Falling back to public interface 2024-02-05T06:05:35.255 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 1 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 2 2024-02-05T06:05:35.256 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 2024-02-05T06:05:34.252+0000 7f4b5bac04c0 -1 osd.2 28 log_to_monitors true 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 3 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:35.257 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:05:35.968 INFO:tasks.workunit.client.0.smithi188.stderr:466793: osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6818/2920547477,v1:127.0.0.1:6819/2920547477] [v2:127.0.0.1:6820/2920547477,v1:127.0.0.1:6821/2920547477] exists,up 74ae8727-9d4d-490e-832c-755a695010d4 2024-02-05T06:05:35.968 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:05:35.968 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:05:35.968 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:05:35.969 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 1 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 2 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: 3' 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-02-05T06:05:35.970 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770' 2024-02-05T06:05:35.971 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:35.971 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:05:35.971 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-02-05T06:05:35.971 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-02-05T06:05:35.971 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770 1-42949672985' 2024-02-05T06:05:35.971 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tlean_interval [15,28) [v2:127.0.0.1:6826/276568886,v1:127.0.0.1:6827/276568886] [v2:127.0.0.1:6828/276568886,v1:127.0.0.1:6829/276568886] exists,up 29a22b72-7c0d-4831-8828-e2d38da57c07 2024-02-05T06:05:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:05:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:05:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:05:36.033 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:05:36.034 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 1 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 2 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: 3' 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920772 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920772 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772' 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:05:36.035 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772 1-42949672987' 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph telllean_interval [20,28) [v2:127.0.0.1:6802/3643052669,v1:127.0.0.1:6803/3643052669] [v2:127.0.0.1:6804/3643052669,v1:127.0.0.1:6805/3643052669] exists,up 1de05a9d-1ae6-4d26-ad37-ca2861ec581c 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:05:36.036 INFO:tasks.workunit.client.0.smithi188.stderr:466796: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 1 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 2 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: 3' 2024-02-05T06:05:36.037 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920771 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920771 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771' 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672986 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672986 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771 1-42949672986' 2024-02-05T06:05:36.038 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph ell osd.2 flush_pg_stats 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770 1-42949672985 2-141733920770' 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770 1-42949672985 2-141733920770 3-137438953474' 2024-02-05T06:05:38.140 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-141733920770 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-141733920770 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 141733920770' 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: waiting osd.0 seq 141733920770 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920770 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-02-05T06:05:38.141 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: waiting osd.1 seq 42949672985 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672985 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672985 2024-02-05T06:05:38.142 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.188 INFO:tasks.workunit.client.0.smithi188.stderr:46679 osd.2 flush_pg_stats 2024-02-05T06:05:38.188 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920771 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920771 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772 1-42949672987 2-141733920771' 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953475 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953475 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772 1-42949672987 2-141733920771 3-137438953475' 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-141733920772 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:05:38.189 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-141733920772 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920772 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 141733920772' 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: waiting osd.0 seq 141733920772 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920772 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-02-05T06:05:38.190 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: waiting osd.1 seq 42949672987 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672987 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672987 2024-02-05T06:05:38.191 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466793: tell osd.2 flush_pg_stats 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920772 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920772 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771 1-42949672986 2-141733920772' 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953476 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953476 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771 1-42949672986 2-141733920772 3-137438953476' 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-141733920771 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:38.207 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-141733920771 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920771 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: waiting osd.0 seq 141733920771 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920771 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672986 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672986 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:38.208 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672986 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: waiting osd.1 seq 42949672986 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672986 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672986 2024-02-05T06:05:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:39.404 INFO:tasks.workunit.client.0.smithi188.stderr:46670: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920770 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920770 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: waiting osd.2 seq 141733920770 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920770 2024-02-05T06:05:39.405 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-137438953474 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-137438953474 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 137438953474' 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: waiting osd.3 seq 137438953474 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953474 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:39.406 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:05:39.407 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:39.496 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.cl//home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920771 2024-02-05T06:05:39.496 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:39.496 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:05:39.496 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920771 2024-02-05T06:05:39.496 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:39.496 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920771 2024-02-05T06:05:39.496 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: waiting osd.2 seq 141733920771 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920771 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-137438953475 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-137438953475 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:39.497 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953475 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 137438953475' 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: waiting osd.3 seq 137438953475 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953475 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:39.498 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:39.499 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:39.499 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:39.499 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:39.499 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:39.499 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:05:39.499 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.clien96: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920772 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920772 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920772 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920772' 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: waiting osd.2 seq 141733920772 2024-02-05T06:05:39.500 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920772 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-137438953476 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-137438953476 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953476 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 137438953476' 2024-02-05T06:05:39.501 INFO:tasks.workunit.client.0.smithi188.stderr:466796: waiting osd.3 seq 137438953476 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953476 -lt 137438953476 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:05:39.502 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:05:39.503 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:05:39.503 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:05:39.503 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:05:39.503 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:05:39.752 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.cient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:39.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:39.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:05:39.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:05:39.753 INFO:tasks.workunit.client.0.smithi188.stderr:466790: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:05:39.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:05:39.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:05:39.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 466788 2024-02-05T06:05:39.841 INFO:tasks.workunit.client.0.smithi188.stderr:t.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:39.841 INFO:tasks.workunit.client.0.smithi188.stderr:466793: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:39.841 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:05:39.841 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:05:39.841 INFO:tasks.workunit.client.0.smithi188.stderr:466793: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:05:39.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:05:39.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:05:39.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 466791 2024-02-05T06:05:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:05:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:466796: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:05:39.848 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:05:39.849 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:05:39.849 INFO:tasks.workunit.client.0.smithi188.stderr:466796: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:05:39.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:05:39.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T06:05:39.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T06:05:39.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T06:05:39.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-02-05T06:05:39.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-02-05T06:05:39.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T06:05:39.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:05:39.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T06:05:39.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:05:39.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:05:40.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-02-05T06:05:40.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-02-05T06:05:40.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T06:05:40.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T06:05:40.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:05:40.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:05:40.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:40.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:05:40.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:05:20.767419+0000 2024-02-05T06:05:40.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T06:05:40.655 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T06:05:20.767419+0000 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:05:20.767419+0000 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:05:40.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:05:40.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:40.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:05:40.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:05:20.767419+0000 '>' 2024-02-05T06:05:20.767419+0000 2024-02-05T06:05:40.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:05:41.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:05:41.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:05:41.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:05:41.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:05:41.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:05:41.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:05:41.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:05:42.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:05:41.305362+0000 '>' 2024-02-05T06:05:20.767419+0000 2024-02-05T06:05:42.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:05:42.207 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-02-05T06:05:42.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-02-05T06:05:42.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-02-05T06:05:42.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:05:42.541 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:05:42.541 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:05:42.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-02-05T06:05:42.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-02-05T06:05:42.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-02-05T06:05:42.875 INFO:tasks.workunit.client.0.smithi188.stdout: osd: 4 osds: 4 up (since 8s), 4 in (since 56s) 2024-02-05T06:05:42.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-02-05T06:05:42.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-02-05T06:05:43.210 INFO:tasks.workunit.client.0.smithi188.stdout: osd: 4 osds: 4 up (since 8s), 4 in (since 57s) 2024-02-05T06:05:43.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-02-05T06:05:43.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:05:43.579 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:05:43.579 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:05:43.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:05:43.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:05:43.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:05:43.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:05:43.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:05:43.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:05:43.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:05:43.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:05:43.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:05:43.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:05:43.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:05:43.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:05:43.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:05:43.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:05:43.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:05:43.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:05:43.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:43.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:05:43.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:05:43.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:43.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:05:43.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:05:43.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:05:43.754 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:05:43.754 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:43.754 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:43.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:05:43.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:05:43.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:05:43.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:05:43.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:05:43.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:05:43.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:05:43.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:05:43.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:05:43.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:05:43.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:05:43.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:05:43.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:05:43.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:05:43.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:05:43.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:43.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:05:43.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:05:43.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:05:43.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:05:43.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:05:43.765 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:05:43.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:05:43.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:43.766 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:43.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:05:43.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:05:43.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:05:43.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-02-05T06:05:43.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:05:43.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:43.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:43.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69197 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:907: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:908: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: unfound_erasure_coded: local dir=td/osd-scrub-repair 2024-02-05T06:05:43.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: unfound_erasure_coded: local allow_overwrites=true 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:854: unfound_erasure_coded: local poolname=ecpool 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:855: unfound_erasure_coded: local payload=ABCDEF 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-02-05T06:05:43.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-02-05T06:05:44.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:05:44.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:44.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:44.156 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:44.156 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:44.156 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:44.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:44.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-02-05T06:05:44.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:05:44.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:05:44.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:05:44.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:05:44.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:05:44.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:05:44.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:05:44.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get fsid 2024-02-05T06:05:44.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69197/ceph-mon.a.asok 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:05:44.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69197/ceph-mon.a.asok config get mon_host 2024-02-05T06:05:44.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2024-02-05T06:05:44.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-02-05T06:05:44.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:05:44.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:05:44.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:05:44.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-02-05T06:05:44.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:05:45.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:05:45.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:45.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:45.317 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:45.317 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:45.317 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:45.317 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:45.318 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:05:45.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:05:45.345 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: seq 0 3 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:05:45.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:05:45.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:05:45.352 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:05:45.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2dc652a0-938a-4d40-af35-e5ada6124d69 2024-02-05T06:05:45.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2dc652a0-938a-4d40-af35-e5ada6124d69' 2024-02-05T06:05:45.353 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 2dc652a0-938a-4d40-af35-e5ada6124d69 2024-02-05T06:05:45.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:05:45.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5esBlmb16FxAAVAXHvpC5lgXUbegAPSb+2g== 2024-02-05T06:05:45.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5esBlmb16FxAAVAXHvpC5lgXUbegAPSb+2g=="}' 2024-02-05T06:05:45.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2dc652a0-938a-4d40-af35-e5ada6124d69 -i td/osd-scrub-repair/0/new.json 2024-02-05T06:05:45.573 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:05:45.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-02-05T06:05:45.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5esBlmb16FxAAVAXHvpC5lgXUbegAPSb+2g== --osd-uuid 2dc652a0-938a-4d40-af35-e5ada6124d69 2024-02-05T06:05:45.612 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:45.637+0000 7f59657904c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:45.624 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:45.649+0000 7f59657904c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:45.625 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:45.651+0000 7f59657904c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:45.626 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:45.651+0000 7f59657904c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:05:45.626 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:45.651+0000 7f59657904c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:05:45.626 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:45.651+0000 7f59657904c0 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-02-05T06:05:45.716 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:45.741+0000 7f59657904c0 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-02-05T06:05:51.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-02-05T06:05:51.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:05:51.125 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:05:51.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:05:51.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:05:51.486 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:05:51.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:05:51.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:05:51.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:05:51.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:05:51.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:05:51.513 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:51.537+0000 7fec547374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:51.532 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:51.558+0000 7fec547374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:51.533 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:51.559+0000 7fec547374c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:05:51.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:05:51.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:05:51.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:05:51.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:05:51.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:05:51.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:51.793 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:05:51.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:05:51.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:51.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:52.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:52.068 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:52.094+0000 7fec547374c0 -1 Falling back to public interface 2024-02-05T06:05:53.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:53.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:53.060 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:05:53.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:05:53.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:53.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:53.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:54.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:54.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:54.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:05:54.341 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:05:54.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:54.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:54.482 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:05:54.508+0000 7fec547374c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:05:54.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:55.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:55.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:55.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:05:55.617 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:05:55.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:55.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:55.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:56.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:56.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:56.897 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:05:56.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:05:56.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:56.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:57.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:58.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:58.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:58.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:05:58.181 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:05:58.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:58.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:58.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:05:59.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:05:59.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:05:59.546 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:05:59.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:05:59.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:05:59.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:05:59.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:00.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:00.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:00.827 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:06:00.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:06:00.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:00.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:01.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:02.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:02.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:02.111 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:06:02.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:06:02.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:02.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:02.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:03.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:03.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:03.390 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:06:03.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:06:03.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:03.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:03.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:04.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:04.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:04.668 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:06:04.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:06:04.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:04.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:04.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:05.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:05.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:05.948 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:06:05.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:06:05.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:05.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:06.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:07.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:07.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:07.340 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:06:07.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:06:07.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:07.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:07.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:08.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:08.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:08.750 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:06:08.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:06:08.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:08.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:09.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:10.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:10.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:10.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:06:10.219 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:06:10.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:10.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:10.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:11.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:11.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:11.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:06:11.964 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:06:11.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:11.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:12.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:13.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:13.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:13.551 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:06:13.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:06:13.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:13.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:14.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:15.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:15.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:15.510 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:06:15.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:06:15.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:15.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:16.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:17.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:17.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:17.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:06:17.367 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:06:17.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:17.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:06:17.620 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3973762994,v1:127.0.0.1:6803/3973762994] [v2:127.0.0.1:6804/3973762994,v1:127.0.0.1:6805/3973762994] exists,up 2dc652a0-938a-4d40-af35-e5ada6124d69 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:06:17.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:06:17.626 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:06:17.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:06:17.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-02-05T06:06:17.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:06:17.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fce0defb-f75f-4724-abd9-59178a40cdb9 2024-02-05T06:06:17.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fce0defb-f75f-4724-abd9-59178a40cdb9' 2024-02-05T06:06:17.628 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 fce0defb-f75f-4724-abd9-59178a40cdb9 2024-02-05T06:06:17.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:06:17.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZesBlTbTXJxAA4RPkz8EKUI8It8+KhrQjvg== 2024-02-05T06:06:17.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZesBlTbTXJxAA4RPkz8EKUI8It8+KhrQjvg=="}' 2024-02-05T06:06:17.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fce0defb-f75f-4724-abd9-59178a40cdb9 -i td/osd-scrub-repair/1/new.json 2024-02-05T06:06:18.050 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:06:18.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-02-05T06:06:18.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZesBlTbTXJxAA4RPkz8EKUI8It8+KhrQjvg== --osd-uuid fce0defb-f75f-4724-abd9-59178a40cdb9 2024-02-05T06:06:18.091 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:18.117+0000 7f37aa9e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:18.135 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:18.161+0000 7f37aa9e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:18.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:18.162+0000 7f37aa9e24c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:18.136 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:18.162+0000 7f37aa9e24c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:06:18.137 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:18.162+0000 7f37aa9e24c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:06:18.137 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:18.162+0000 7f37aa9e24c0 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-02-05T06:06:18.210 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:18.236+0000 7f37aa9e24c0 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-02-05T06:06:24.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-02-05T06:06:24.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:06:24.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:06:24.279 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:06:24.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:06:24.621 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:06:24.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:06:24.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:06:24.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:06:24.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:06:24.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:06:24.648 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:24.674+0000 7ff1d0ac34c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:24.652 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:24.676+0000 7ff1d0ac34c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:24.652 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:24.677+0000 7ff1d0ac34c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:24.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:06:24.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:06:24.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:06:24.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:06:24.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:06:24.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:24.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:06:24.923 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:06:24.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:24.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:25.174 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:25.200+0000 7ff1d0ac34c0 -1 Falling back to public interface 2024-02-05T06:06:25.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:26.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:26.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:26.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:06:26.221 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:06:26.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:26.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:26.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:27.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:27.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:27.492 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:06:27.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:06:27.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:27.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:27.651 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:27.677+0000 7ff1d0ac34c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:06:27.858 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:28.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:28.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:28.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:06:28.861 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:06:28.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:28.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:29.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:30.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:30.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:30.141 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:06:30.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:06:30.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:30.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:30.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:31.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:31.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:31.497 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:06:31.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:06:31.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:31.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:31.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:32.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:32.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:32.872 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:06:32.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:06:32.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:32.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:33.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:34.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:34.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:34.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:06:34.151 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:06:34.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:34.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:34.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:35.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:35.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:35.458 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:06:35.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:06:35.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:35.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:35.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:36.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:36.814 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:06:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:06:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:36.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:37.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:38.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:38.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:38.093 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:06:38.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:06:38.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:38.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:38.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:39.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:39.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:39.451 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:06:39.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:06:39.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:39.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:40.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:41.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:41.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:41.396 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:06:41.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:06:41.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:41.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:42.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:43.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:43.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:43.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:06:43.223 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:06:43.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:43.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:43.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:44.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:44.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:44.726 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:06:44.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:06:44.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:44.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:45.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:46.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:46.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:46.446 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:06:46.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:06:46.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:46.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:46.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:47.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:47.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:47.931 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:06:47.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:06:47.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:47.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:48.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:49.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:49.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:49.424 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:06:49.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:06:49.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:49.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:50.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:51.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:51.049 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:06:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:06:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:51.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:06:51.325 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/340773084,v1:127.0.0.1:6811/340773084] [v2:127.0.0.1:6812/340773084,v1:127.0.0.1:6813/340773084] exists,up fce0defb-f75f-4724-abd9-59178a40cdb9 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:06:51.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:06:51.331 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:06:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:06:51.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:06:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=402b72a1-2cd5-4f83-a47f-3853e95b5ee4 2024-02-05T06:06:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 402b72a1-2cd5-4f83-a47f-3853e95b5ee4' 2024-02-05T06:06:51.333 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 402b72a1-2cd5-4f83-a47f-3853e95b5ee4 2024-02-05T06:06:51.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:06:51.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7esBlHkZFFhAA1CID7n1BEFI2E+eLpHRkHQ== 2024-02-05T06:06:51.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7esBlHkZFFhAA1CID7n1BEFI2E+eLpHRkHQ=="}' 2024-02-05T06:06:51.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 402b72a1-2cd5-4f83-a47f-3853e95b5ee4 -i td/osd-scrub-repair/2/new.json 2024-02-05T06:06:51.688 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:06:51.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-02-05T06:06:51.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7esBlHkZFFhAA1CID7n1BEFI2E+eLpHRkHQ== --osd-uuid 402b72a1-2cd5-4f83-a47f-3853e95b5ee4 2024-02-05T06:06:51.729 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:51.755+0000 7ff519a1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:51.773 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:51.799+0000 7ff519a1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:51.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:51.800+0000 7ff519a1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:51.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:51.801+0000 7ff519a1a4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:06:51.775 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:51.801+0000 7ff519a1a4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:06:51.775 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:51.801+0000 7ff519a1a4c0 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-02-05T06:06:51.848 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:51.874+0000 7ff519a1a4c0 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-02-05T06:06:57.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-02-05T06:06:57.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:06:57.795 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T06:06:57.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:06:57.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:06:58.141 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T06:06:58.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:06:58.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:06:58.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:06:58.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:06:58.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:06:58.168 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:58.194+0000 7f6e2f0104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:58.170 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:58.196+0000 7f6e2f0104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:58.171 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:58.197+0000 7f6e2f0104c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:06:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:06:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:06:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:06:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:06:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:06:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:58.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:06:58.409 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:06:58.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:58.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:06:58.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:06:58.697 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:06:58.723+0000 7f6e2f0104c0 -1 Falling back to public interface 2024-02-05T06:06:59.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:06:59.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:06:59.674 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:06:59.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:06:59.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:06:59.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:06:59.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:00.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:00.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:00.945 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:07:00.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:07:00.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:00.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:01.030 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:01.056+0000 7f6e2f0104c0 -1 osd.2 0 log_to_monitors true 2024-02-05T06:07:01.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:02.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:02.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:02.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:07:02.248 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:07:02.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:02.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:02.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:03.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:03.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:03.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:07:03.525 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:07:03.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:03.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:03.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:04.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:04.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:04.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:07:04.993 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:07:04.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:04.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:05.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:06.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:06.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:06.338 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:07:06.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:07:06.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:06.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:06.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:07.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:07.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:07.635 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:07:07.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:07:07.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:07.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:07.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:08.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:08.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:08.912 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:07:08.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:07:08.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:08.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:09.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:10.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:10.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:10.190 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:07:10.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:07:10.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:10.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:10.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:11.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:11.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:11.466 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:07:11.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:07:11.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:11.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:11.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:12.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:12.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:12.868 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:07:12.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:07:12.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:12.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:13.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:14.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:14.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:14.146 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:07:14.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:07:14.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:14.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:14.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:15.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:15.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:15.842 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:07:15.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:07:15.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:15.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:16.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:17.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:17.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:17.292 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:07:17.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:07:17.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:17.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:17.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:18.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:18.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:18.753 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:07:18.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:07:18.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:18.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:19.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:20.315 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:20.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:20.316 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:07:20.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:07:20.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:20.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:20.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:21.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:21.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:21.901 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:07:21.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:07:21.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:21.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:22.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:23.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:23.265 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:23.265 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:07:23.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:07:23.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:23.266 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:24.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:25.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:25.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:25.081 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:07:25.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:07:25.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:25.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:07:25.358 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/93487433,v1:127.0.0.1:6819/93487433] [v2:127.0.0.1:6820/93487433,v1:127.0.0.1:6821/93487433] exists,up 402b72a1-2cd5-4f83-a47f-3853e95b5ee4 2024-02-05T06:07:25.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:07:25.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:07:25.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:07:25.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: unfound_erasure_coded: for id in $(seq 0 3) 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:860: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:07:25.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:07:25.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:07:25.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14d25ee8-a3fe-436a-b024-9bfbca9bc381 2024-02-05T06:07:25.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 14d25ee8-a3fe-436a-b024-9bfbca9bc381' 2024-02-05T06:07:25.366 INFO:tasks.workunit.client.0.smithi188.stdout:add osd3 14d25ee8-a3fe-436a-b024-9bfbca9bc381 2024-02-05T06:07:25.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:07:25.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAde8BlSf01GBAADjV0dDBSaX5Ko+FDOCYUsQ== 2024-02-05T06:07:25.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAde8BlSf01GBAADjV0dDBSaX5Ko+FDOCYUsQ=="}' 2024-02-05T06:07:25.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14d25ee8-a3fe-436a-b024-9bfbca9bc381 -i td/osd-scrub-repair/3/new.json 2024-02-05T06:07:25.739 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:07:25.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-02-05T06:07:25.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAde8BlSf01GBAADjV0dDBSaX5Ko+FDOCYUsQ== --osd-uuid 14d25ee8-a3fe-436a-b024-9bfbca9bc381 2024-02-05T06:07:25.780 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:25.806+0000 7f0b3880c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:25.832 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:25.858+0000 7f0b3880c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:25.833 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:25.859+0000 7f0b3880c4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:25.833 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:25.860+0000 7f0b3880c4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:07:25.833 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:25.860+0000 7f0b3880c4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:07:25.833 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:25.860+0000 7f0b3880c4c0 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-02-05T06:07:25.907 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:25.934+0000 7f0b3880c4c0 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-02-05T06:07:31.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-02-05T06:07:31.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:07:31.640 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd3 key to auth repository 2024-02-05T06:07:31.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-02-05T06:07:31.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:07:31.989 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.3 2024-02-05T06:07:31.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-02-05T06:07:31.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:07:31.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:07:31.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:07:31.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:07:32.014 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:32.041+0000 7f7ad4f1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:32.016 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:32.042+0000 7f7ad4f1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:32.017 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:32.043+0000 7f7ad4f1a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:07:32.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-02-05T06:07:32.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:07:32.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:07:32.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:07:32.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:07:32.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:32.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:07:32.260 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:07:32.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:32.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:32.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:32.541 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:32.568+0000 7f7ad4f1a4c0 -1 Falling back to public interface 2024-02-05T06:07:33.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:33.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:33.526 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:07:33.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:07:33.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:33.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:33.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:34.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:34.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:34.805 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:07:34.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:07:34.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:34.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:34.881 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:07:34.908+0000 7f7ad4f1a4c0 -1 osd.3 0 log_to_monitors true 2024-02-05T06:07:35.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:36.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:36.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:36.084 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:07:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:07:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:36.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:36.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:37.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:37.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:37.368 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:07:37.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:07:37.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:37.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:37.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:38.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:38.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:38.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:07:38.661 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:07:38.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:38.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:38.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:39.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:39.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:39.939 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:07:39.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:07:39.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:39.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:40.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:41.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:41.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:41.214 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:07:41.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:07:41.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:41.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:41.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:42.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:42.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:42.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:07:42.495 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:07:42.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:42.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:42.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:43.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:43.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:43.894 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:07:43.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:07:43.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:43.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:44.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:45.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:45.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:45.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:07:45.174 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:07:45.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:45.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:45.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:46.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:46.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:46.532 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:07:46.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:07:46.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:46.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:46.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:47.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:47.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:47.932 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:07:47.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:07:47.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:47.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:48.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:49.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:49.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:49.300 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:07:49.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:07:49.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:49.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:49.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:50.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:50.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:50.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:07:50.827 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:07:50.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:50.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:51.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:52.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:52.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:52.681 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:07:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:07:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:52.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:53.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:54.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:54.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:54.330 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:07:54.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:07:54.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:54.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:54.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:55.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:55.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:55.625 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:07:55.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:07:55.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:55.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:56.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:57.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:57.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:57.395 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:07:57.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:07:57.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:57.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:57.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:07:58.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:07:58.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:07:58.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:07:58.686 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:07:58.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:07:58.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:07:58.960 INFO:tasks.workunit.client.0.smithi188.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2750705644,v1:127.0.0.1:6827/2750705644] [v2:127.0.0.1:6828/2750705644,v1:127.0.0.1:6829/2750705644] exists,up 14d25ee8-a3fe-436a-b024-9bfbca9bc381 2024-02-05T06:07:58.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-02-05T06:07:58.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-02-05T06:07:59.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-02-05T06:07:59.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-02-05T06:08:00.063 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'ecpool' created 2024-02-05T06:08:00.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:08:01.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-02-05T06:08:01.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-02-05T06:08:01.686 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 allow_ec_overwrites to true 2024-02-05T06:08:01.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-02-05T06:08:01.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:01.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:01.697 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:01.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:01.698 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:01.698 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:01.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:01.698 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:01.699 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:01.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:01.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:01.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:01.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:01.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:01.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:02.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:02.057 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:08:02.058 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T06:08:02.058 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T06:08:02.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:02.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:02.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:02.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-02-05T06:08:02.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-02-05T06:08:02.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-02-05T06:08:02.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:02.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:02.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-02-05T06:08:02.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-02-05T06:08:02.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672978' 2024-02-05T06:08:02.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:02.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:08:02.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-02-05T06:08:02.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-02-05T06:08:02.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672978 2-64424509453' 2024-02-05T06:08:02.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:02.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:08:02.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345929 2024-02-05T06:08:02.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345929 2024-02-05T06:08:02.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672978 2-64424509453 3-85899345929' 2024-02-05T06:08:02.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:02.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-02-05T06:08:02.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:02.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:02.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-02-05T06:08:02.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:02.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-02-05T06:08:02.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-02-05T06:08:02.558 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836506 2024-02-05T06:08:02.558 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:02.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836506 2024-02-05T06:08:02.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:03.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:03.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:04.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-02-05T06:08:04.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:04.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-02-05T06:08:04.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:04.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:04.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-02-05T06:08:04.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:04.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-02-05T06:08:04.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-02-05T06:08:04.111 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672978 2024-02-05T06:08:04.111 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:04.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-02-05T06:08:04.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:04.379 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-02-05T06:08:04.379 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:04.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:08:04.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-02-05T06:08:04.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:04.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-02-05T06:08:04.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-02-05T06:08:04.383 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509453 2024-02-05T06:08:04.383 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:08:04.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-02-05T06:08:04.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:04.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345929 2024-02-05T06:08:04.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:04.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:08:04.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345929 2024-02-05T06:08:04.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:04.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345929 2024-02-05T06:08:04.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-02-05T06:08:04.654 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.3 seq 85899345929 2024-02-05T06:08:04.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:08:04.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345929 -lt 85899345929 2024-02-05T06:08:04.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:04.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:04.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:05.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:08:05.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:05.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:05.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:05.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:05.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:05.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:05.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:05.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:08:05.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:05.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:05.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:05.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:865: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-02-05T06:08:05.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-02-05T06:08:06.209 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:08:06.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-02-05T06:08:06.728 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:08:06.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-02-05T06:08:06.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-02-05T06:08:06.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-02-05T06:08:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: get_primary ecpool SOMETHING 2024-02-05T06:08:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-02-05T06:08:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-02-05T06:08:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:08:06.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:08:07.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: unfound_erasure_coded: local primary=1 2024-02-05T06:08:07.285 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: get_osds ecpool SOMETHING 2024-02-05T06:08:07.286 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-02-05T06:08:07.286 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: sed -e s/1// 2024-02-05T06:08:07.286 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-02-05T06:08:07.286 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:08:07.286 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:0 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:2 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:3' 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 3 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: osds=('0' '2' '3') 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:868: unfound_erasure_coded: local -a osds 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: unfound_erasure_coded: local not_primary_first=0 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: unfound_erasure_coded: local not_primary_second=2 2024-02-05T06:08:07.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:871: unfound_erasure_coded: local not_primary_third=3 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:876: unfound_erasure_coded: pids= 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:877: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 478834"' 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 478834' 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 478835"' 2024-02-05T06:08:07.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 478835' 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/478837: /' 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 478838"' 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 478838' 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:880: unfound_erasure_coded: wait_background pids 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 478834 478835 478838' 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 478834 2024-02-05T06:08:07.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:08:07.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/478840: /' 2024-02-05T06:08:07.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:08:07.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/478843: /' 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-02-05T06:08:10.581 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: remove 1#1:eb822e21:::SOMETHING:head# 2024-02-05T06:08:10.582 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-02-05T06:08:10.583 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.583 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:10.583 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:08:10.583 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:10.583 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-02-05T06:08:10.583 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:10.583 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:10.584 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:10.584 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:10.584 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-02-05T06:08:10.584 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:10.585 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: start osd.0 2024-02-05T06:08:10.586 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-02-05T06:08:10.623 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:08:10.624 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: remove 3#1:eb822e21:::SOMETHING:head# 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: 478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.625 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:10.626 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: remove 2#1:eb822e21:::SOMETHING:head# 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-02-05T06:08:10.627 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:10.628 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:10.628 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:10.628 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-02-05T06:08:10.628 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-02-05T06:08:10.628 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:10.628 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:10.629 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: start osd.2 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-maxactivate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-02-05T06:08:10.630 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:10.631 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: start osd.3 2024-02-05T06:08:10.632 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=d97a0627-2f91-4152-81aa-0108a1689cc6 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.69197/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:15.282 INFO:tasks.workunit.client.0.smithi188.stderr:-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:15.282 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-02-05T06:08:15.282 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:15.282 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:15.282 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:15.282 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 2024-02-05T06:08:10.700+0000 7f76d44da4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 2024-02-05T06:08:10.703+0000 7f76d44da4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 2024-02-05T06:08:10.705+0000 7f76d44da4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 0 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.283 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 2024-02-05T06:08:11.231+0000 7f76d44da4c0 -1 Falling back to public interface 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 1 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:15.284 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 2 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 2024-02-05T06:08:14.205+0000 7f76d44da4c0 -1 osd.2 29 log_to_monitors true 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 3 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.285 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:08:15.287 INFO:tasks.workunit.client.0.smithi188.stderr:478840: osd.2 up in weight 1 up_from 34 up_thru 0 down_at 30 last_c478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-02-05T06:08:15.287 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:08:15.287 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:15.287 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:15.287 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 2024-02-05T06:08:10.640+0000 7ff4ab1944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 2024-02-05T06:08:10.643+0000 7ff4ab1944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 2024-02-05T06:08:10.644+0000 7ff4ab1944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 0 2024-02-05T06:08:15.288 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 2024-02-05T06:08:11.160+0000 7ff4ab1944c0 -1 Falling back to public interface 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 1 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 2 2024-02-05T06:08:15.289 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 2024-02-05T06:08:13.954+0000 7ff4ab1944c0 -1 osd.0 29 log_to_monitors true 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 3 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478837: osd.0 up in weight 1 up_from 33 up_thru 0 down_at 30 last_c478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-02-05T06:08:15.290 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 2024-02-05T06:08:10.682+0000 7f3d584164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 2024-02-05T06:08:10.685+0000 7f3d584164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 2024-02-05T06:08:10.686+0000 7f3d584164c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.291 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 0 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 2024-02-05T06:08:11.201+0000 7f3d584164c0 -1 Falling back to public interface 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.292 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 1 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 2 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 2024-02-05T06:08:14.130+0000 7f3d584164c0 -1 osd.3 29 log_to_monitors true 2024-02-05T06:08:15.293 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:15.294 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:15.294 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:15.294 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 3 2024-02-05T06:08:15.294 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:15.294 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478843: osd.3 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/678980054,v1:127.0.0.1:6803/678980054] [v2:127.0.0.1:6804/678980054,v1:127.0.0.1:6805/678980054] exists,up 2dc652a0-938a-4d40-af35-e5ada6124d69 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:478837: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 1 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 2 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: 3' 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.006 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920770 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920770 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770' 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770 1-42949672981' 2024-02-05T06:08:16.007 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.057 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell lean_interval [15,29) [v2:127.0.0.1:6826/4058876705,v1:127.0.0.1:6827/4058876705] [v2:127.0.0.1:6828/4058876705,v1:127.0.0.1:6829/4058876705] exists,up 402b72a1-2cd5-4f83-a47f-3853e95b5ee4 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:16.058 INFO:tasks.workunit.client.0.smithi188.stderr:478840: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 1 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 2 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: 3' 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:16.059 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920771 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920771 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771' 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672982 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672982 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771 1-42949672982' 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph lean_interval [20,29) [v2:127.0.0.1:6818/3775606373,v1:127.0.0.1:6819/3775606373] [v2:127.0.0.1:6820/3775606373,v1:127.0.0.1:6821/3775606373] exists,up 14d25ee8-a3fe-436a-b024-9bfbca9bc381 2024-02-05T06:08:16.060 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:08:16.061 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 1 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 2 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: 3' 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920772 2024-02-05T06:08:16.062 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920772 2024-02-05T06:08:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772' 2024-02-05T06:08:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:08:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-02-05T06:08:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-02-05T06:08:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772 1-42949672983' 2024-02-05T06:08:16.063 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:18.116 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph osd.2 flush_pg_stats 2024-02-05T06:08:18.116 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770 1-42949672981 2-146028888066' 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920770 1-42949672981 2-146028888066 3-146028888066' 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-141733920770 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-141733920770 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920770 2024-02-05T06:08:18.117 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 141733920770' 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: waiting osd.0 seq 141733920770 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 141733920770 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920770 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-02-05T06:08:18.118 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-02-05T06:08:18.119 INFO:tasks.workunit.client.0.smithi188.stderr:478837: waiting osd.1 seq 42949672981 2024-02-05T06:08:18.119 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:18.119 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672981 2024-02-05T06:08:18.119 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.194 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubutell osd.2 flush_pg_stats 2024-02-05T06:08:18.194 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-02-05T06:08:18.194 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771 1-42949672982 2-146028888068' 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920771 1-42949672982 2-146028888068 3-146028888067' 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-141733920771 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-141733920771 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920771 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2024-02-05T06:08:18.195 INFO:tasks.workunit.client.0.smithi188.stderr:478840: waiting osd.0 seq 141733920771 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 141733920771 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920771 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672982 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672982 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672982 2024-02-05T06:08:18.196 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478840: waiting osd.1 seq 42949672982 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672982 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //homtell osd.2 flush_pg_stats 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888067 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888067 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772 1-42949672983 2-146028888067' 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888068 2024-02-05T06:08:18.197 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888068 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-141733920772 1-42949672983 2-146028888067 3-146028888068' 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-141733920772 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-141733920772 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920772 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 141733920772' 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: waiting osd.0 seq 141733920772 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 141733920772 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:08:18.198 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:08:18.199 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:08:18.199 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920772 -lt 141733920772 2024-02-05T06:08:18.199 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:18.199 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-02-05T06:08:18.199 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:18.199 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:08:18.199 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-02-05T06:08:18.200 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:18.200 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-02-05T06:08:18.200 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-02-05T06:08:18.200 INFO:tasks.workunit.client.0.smithi188.stderr:478843: waiting osd.1 seq 42949672983 2024-02-05T06:08:18.200 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:08:18.200 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672983 2024-02-05T06:08:18.201 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:19.419 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //homntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888066 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888066 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888066' 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: waiting osd.2 seq 146028888066 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888066 2024-02-05T06:08:19.420 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888066 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888066 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888066' 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: waiting osd.3 seq 146028888066 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888066 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:19.421 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:08:19.422 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:19.497 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/stae/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888068 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888068 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888068' 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: waiting osd.2 seq 146028888068 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888067 2024-02-05T06:08:19.498 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888067 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888067' 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: waiting osd.3 seq 146028888067 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:19.499 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:08:19.500 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:19.505 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qe/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-146028888067 2024-02-05T06:08:19.505 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:19.505 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:08:19.505 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-146028888067 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888067 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: waiting osd.2 seq 146028888067 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888067 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-146028888068 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:08:19.506 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-146028888068 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888068 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 146028888068' 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: waiting osd.3 seq 146028888068 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888068 -lt 146028888068 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:08:19.507 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:08:19.508 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:08:19.508 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:08:19.508 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:08:19.508 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:08:19.508 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:08:19.815 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qndalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi188.stderr:478837: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi188.stderr:478837: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 478835 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:a/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:478840: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:478840: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-02-05T06:08:19.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 478838 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:a/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:478843: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:478843: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-02-05T06:08:19.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:881: unfound_erasure_coded: return_code=0 2024-02-05T06:08:19.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:882: unfound_erasure_coded: '[' 0 -ne 0 ']' 2024-02-05T06:08:19.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: get_pg ecpool SOMETHING 2024-02-05T06:08:19.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-02-05T06:08:19.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-02-05T06:08:19.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-02-05T06:08:19.888 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-02-05T06:08:20.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:887: unfound_erasure_coded: local pg=1.0 2024-02-05T06:08:20.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: unfound_erasure_coded: repair 1.0 2024-02-05T06:08:20.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-02-05T06:08:20.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-02-05T06:08:20.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:08:20.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:08:20.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:08:20.158 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:08:20.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:20.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-02-05T06:08:20.687 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-02-05T06:08:20.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:20.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:08:20.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:20.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:08:20.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:08:20.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:08:20.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:08:20.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:08:20.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:08:20.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:08:20.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:08:20.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:59.981290+0000 '>' 2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:20.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:08:21.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:08:21.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:08:21.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:08:21.964 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:08:21.965 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:08:21.965 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:08:21.965 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:08:22.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:59.981290+0000 '>' 2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:22.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:08:23.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:08:23.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:08:23.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:08:23.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:08:23.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:08:23.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:08:23.240 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:08:23.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:59.981290+0000 '>' 2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:23.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:08:24.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:08:24.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:08:24.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:08:24.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:08:24.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:08:24.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:08:24.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:08:24.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:59.981290+0000 '>' 2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:24.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:08:25.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:08:25.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:08:25.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:08:25.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:08:25.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:08:25.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:08:25.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:08:26.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:07:59.981290+0000 '>' 2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:26.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:08:27.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:08:27.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:08:27.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:08:27.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:08:27.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:08:27.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:08:27.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:08:27.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:08:21.701295+0000 '>' 2024-02-05T06:07:59.981290+0000 2024-02-05T06:08:27.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:08:27.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: seq 1 60 2024-02-05T06:08:27.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: unfound_erasure_coded: for f in `seq 1 60` 2024-02-05T06:08:27.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: ceph -s 2024-02-05T06:08:27.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:08:27.692 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:08:27.692 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:08:27.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: unfound_erasure_coded: break 2024-02-05T06:08:27.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: ceph -s 2024-02-05T06:08:27.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: unfound_erasure_coded: grep '4 up' 2024-02-05T06:08:28.021 INFO:tasks.workunit.client.0.smithi188.stdout: osd: 4 osds: 4 up (since 13s), 4 in (since 62s) 2024-02-05T06:08:28.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: ceph -s 2024-02-05T06:08:28.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:898: unfound_erasure_coded: grep '4 in' 2024-02-05T06:08:28.441 INFO:tasks.workunit.client.0.smithi188.stdout: osd: 4 osds: 4 up (since 13s), 4 in (since 62s) 2024-02-05T06:08:28.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: ceph -s 2024-02-05T06:08:28.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:899: unfound_erasure_coded: grep '1/1 objects unfound' 2024-02-05T06:08:28.771 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:08:28.771 INFO:tasks.workunit.client.0.smithi188.stdout: 1/1 objects unfound (100.000%) 2024-02-05T06:08:28.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-02-05T06:08:28.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:08:28.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:08:28.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:08:28.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:28.773 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:28.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:28.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:28.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:28.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:28.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:08:28.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:08:28.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:08:28.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:08:28.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:08:28.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:08:28.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:08:28.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:08:28.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:08:28.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:08:28.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:08:28.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:08:28.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:08:28.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:08:28.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:28.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:08:28.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:08:28.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 0 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:28.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:28.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:28.913 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:08:28.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:08:28.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:08:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:08:28.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:08:28.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:08:28.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:08:28.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:08:28.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:08:28.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:08:28.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:08:28.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T06:08:28.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-02-05T06:08:28.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:08:28.920 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:28.921 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69197 2024-02-05T06:08:28.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69197 2024-02-05T06:08:28.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:08:28.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:08:28.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T06:08:28.923 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T06:08:28.923 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T06:08:28.990 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2024-02-05T06:08:28.991 DEBUG:teuthology.orchestra.run.smithi188:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2024-02-05T06:08:29.051 INFO:tasks.workunit.client.0.smithi188.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T06:08:29.057 INFO:tasks.workunit.client.0.smithi188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T06:08:29.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T06:08:29.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-snaps 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2024-02-05T06:08:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none ' 2024-02-05T06:08:29.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2024-02-05T06:08:29.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T06:08:29.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2024-02-05T06:08:29.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_scrub_snaps_primary 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_scrub_snaps_replica' 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:08:29.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:08:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:08:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:08:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:08:29.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:08:29.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:08:29.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:08:29.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:08:29.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:08:29.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:08:29.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:08:29.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:08:29.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:08:29.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:08:29.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:08:29.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:08:29.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:08:29.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:08:29.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:29.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:08:29.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.483225 2024-02-05T06:08:29.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:08:29.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:08:29.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-02-05T06:08:29.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:08:29.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:29.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:08:29.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.483225 2024-02-05T06:08:29.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:08:29.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T06:08:29.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T06:08:29.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:08:29.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.483225 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-02-05T06:08:29.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2024-02-05T06:08:29.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:08:29.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:29.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:29.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:29.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:29.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:08:29.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:08:29.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2024-02-05T06:08:29.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:08:29.631 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:08:29.632 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:29.632 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:08:29.632 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.483225/ceph-mon.a.asok 2024-02-05T06:08:29.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:08:29.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.483225/ceph-mon.a.asok config get fsid 2024-02-05T06:08:29.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:08:29.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:08:29.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:08:29.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:08:29.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:08:29.710 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.483225/ceph-mon.a.asok 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:08:29.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.483225/ceph-mon.a.asok config get mon_host 2024-02-05T06:08:29.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2024-02-05T06:08:29.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-02-05T06:08:29.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:08:29.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:08:29.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:08:29.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-02-05T06:08:29.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:08:30.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:08:30.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:30.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:30.591 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:30.591 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:30.591 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:08:30.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:08:30.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:08:30.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:08:30.616 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2024-02-05T06:08:30.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2024-02-05T06:08:30.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:08:30.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2024-02-05T06:08:30.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:08:30.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:08:30.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:08:30.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:08:30.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:08:30.622 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:08:30.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:08:30.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:08:30.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7575750e-c9cf-4e21-ae7c-a91d34c20027 2024-02-05T06:08:30.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7575750e-c9cf-4e21-ae7c-a91d34c20027' 2024-02-05T06:08:30.624 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 7575750e-c9cf-4e21-ae7c-a91d34c20027 2024-02-05T06:08:30.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:08:30.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBee8BlvV6cJxAA7y4ZDNjrz3WgwuNlPc41Kg== 2024-02-05T06:08:30.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBee8BlvV6cJxAA7y4ZDNjrz3WgwuNlPc41Kg=="}' 2024-02-05T06:08:30.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7575750e-c9cf-4e21-ae7c-a91d34c20027 -i td/osd-scrub-snaps/0/new.json 2024-02-05T06:08:30.838 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:08:30.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-02-05T06:08:30.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBee8BlvV6cJxAA7y4ZDNjrz3WgwuNlPc41Kg== --osd-uuid 7575750e-c9cf-4e21-ae7c-a91d34c20027 2024-02-05T06:08:30.874 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:30.901+0000 7f5c48ff94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:30.889 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:30.916+0000 7f5c48ff94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:30.890 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:30.917+0000 7f5c48ff94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:30.890 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:30.918+0000 7f5c48ff94c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:08:30.891 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:30.918+0000 7f5c48ff94c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:08:30.891 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:30.918+0000 7f5c48ff94c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:08:30.973 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:31.000+0000 7f5c48ff94c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-02-05T06:08:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-02-05T06:08:36.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:08:36.350 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:08:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:08:36.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:08:36.699 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:08:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:08:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:08:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:08:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:08:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:08:36.723 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:36.750+0000 7fcf48e0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:36.749 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:36.776+0000 7fcf48e0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:36.750 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:36.778+0000 7fcf48e0d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:08:36.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:08:36.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:08:36.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:08:36.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:08:36.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:08:36.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:36.969 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:08:36.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:08:36.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:36.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:37.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:37.281 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:37.308+0000 7fcf48e0d4c0 -1 Falling back to public interface 2024-02-05T06:08:38.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:38.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:38.249 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:08:38.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:08:38.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:38.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:38.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:39.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:39.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:39.527 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:08:39.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:08:39.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:39.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:39.598 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:08:39.626+0000 7fcf48e0d4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:08:39.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:40.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:40.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:40.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:08:40.799 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:08:40.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:40.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:41.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:42.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:42.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:42.077 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:08:42.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:08:42.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:42.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:42.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:43.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:43.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:43.394 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:08:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:08:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:43.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:44.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:44.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:44.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:08:44.667 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:08:44.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:44.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:44.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:45.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:45.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:45.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:08:45.944 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:08:45.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:45.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:46.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:47.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:47.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:47.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:08:47.219 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:08:47.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:47.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:47.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:48.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:48.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:48.559 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:08:48.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:08:48.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:48.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:48.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:49.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:49.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:49.834 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:08:49.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:08:49.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:49.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:50.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:51.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:51.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:51.159 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:08:51.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:08:51.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:51.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:51.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:52.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:52.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:52.924 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:08:52.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:08:52.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:52.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:53.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:54.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:54.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:54.571 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:08:54.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:08:54.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:54.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:55.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:56.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:56.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:56.319 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:08:56.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:08:56.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:56.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:56.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:57.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:57.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:57.805 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:08:57.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:08:57.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:57.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:58.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:08:59.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:08:59.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:08:59.242 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:08:59.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:08:59.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:08:59.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:08:59.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:00.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:00.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:00.550 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:09:00.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:09:00.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:00.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:01.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:02.396 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:09:02.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:02.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:02.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:09:02.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:02.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:02.715 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3433310917,v1:127.0.0.1:6803/3433310917] [v2:127.0.0.1:6804/3433310917,v1:127.0.0.1:6805/3433310917] exists,up 7575750e-c9cf-4e21-ae7c-a91d34c20027 2024-02-05T06:09:02.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:09:02.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:09:02.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:09:02.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2024-02-05T06:09:03.121 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:09:03.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2024-02-05T06:09:03.498 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:09:03.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2024-02-05T06:09:03.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:09:03.951 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T06:09:03.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:09:04.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2024-02-05T06:09:04.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:09:04.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:09:04.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:09:04.965 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:09:04.966 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:09:04.966 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:09:04.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:09:04.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:09:04.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:09:05.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:09:05.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:09:05.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:09:05.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:09:05.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:09:05.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:09:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:09:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:09:05.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:09:05.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:09:05.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836483 2024-02-05T06:09:05.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836483 2024-02-05T06:09:05.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836483' 2024-02-05T06:09:05.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:09:05.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836483 2024-02-05T06:09:05.420 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:09:05.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:09:05.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836483 2024-02-05T06:09:05.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:09:05.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836483 2024-02-05T06:09:05.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2024-02-05T06:09:05.423 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836483 2024-02-05T06:09:05.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:05.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836482 -lt 21474836483 2024-02-05T06:09:05.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:09:06.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:09:06.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:09:06.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836483 -lt 21474836483 2024-02-05T06:09:06.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:09:06.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:06.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:07.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:09:07.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:09:07.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:09:07.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:09:07.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:09:07.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:09:07.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:09:07.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:09:07.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:09:07.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:09:07.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:09:07.573 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:09:07.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:09:07.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:09:07.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:09:07.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2024-02-05T06:09:07.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2024-02-05T06:09:07.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2024-02-05T06:09:08.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2024-02-05T06:09:08.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.483225 bs=1032 count=1 2024-02-05T06:09:08.170 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:09:08.170 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:09:08.170 INFO:tasks.workunit.client.0.smithi188.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.4586e-05 s, 23.1 MB/s 2024-02-05T06:09:08.170 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2024-02-05T06:09:08.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.483225 2024-02-05T06:09:08.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.483225 2024-02-05T06:09:08.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.483225 2024-02-05T06:09:08.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.483225 2024-02-05T06:09:08.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.483225 2024-02-05T06:09:08.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.483225 2024-02-05T06:09:08.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.483225 2024-02-05T06:09:08.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.483225 2024-02-05T06:09:08.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:08.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.483225 2024-02-05T06:09:09.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:09.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.483225 2024-02-05T06:09:09.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:09.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.483225 2024-02-05T06:09:09.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:09.169 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.483225 2024-02-05T06:09:09.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:09.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.483225 2024-02-05T06:09:09.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:09.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.483225 2024-02-05T06:09:09.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:09.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.483225 2024-02-05T06:09:09.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2024-02-05T06:09:09.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.483225 2024-02-05T06:09:09.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2024-02-05T06:09:09.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:09:09.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:09:09.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:09:09.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.483225 0 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.483225 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-02-05T06:09:09.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-02-05T06:09:10.029 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap1 2024-02-05T06:09:10.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=1 2024-02-05T06:09:10.032 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:09:10.032 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:09:10.032 INFO:tasks.workunit.client.0.smithi188.stderr:256 bytes copied, 5.9954e-05 s, 4.3 MB/s 2024-02-05T06:09:10.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.483225 2024-02-05T06:09:10.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:09:10.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.483225 2024-02-05T06:09:10.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-02-05T06:09:10.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:10.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.483225 2024-02-05T06:09:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:10.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.483225 2024-02-05T06:09:10.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:10.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.483225 2024-02-05T06:09:10.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:10.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.483225 2024-02-05T06:09:10.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:10.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.483225 2024-02-05T06:09:10.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:10.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.483225 2024-02-05T06:09:11.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:11.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.483225 2024-02-05T06:09:11.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:11.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.483225 2024-02-05T06:09:11.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:09:11.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.483225 2024-02-05T06:09:11.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-02-05T06:09:11.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-02-05T06:09:11.378 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap2 2024-02-05T06:09:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=2 2024-02-05T06:09:11.381 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records in 2024-02-05T06:09:11.381 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records out 2024-02-05T06:09:11.381 INFO:tasks.workunit.client.0.smithi188.stderr:512 bytes copied, 6.067e-05 s, 8.4 MB/s 2024-02-05T06:09:11.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:09:11.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-02-05T06:09:11.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-02-05T06:09:11.755 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap3 2024-02-05T06:09:11.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=3 2024-02-05T06:09:11.758 INFO:tasks.workunit.client.0.smithi188.stderr:3+0 records in 2024-02-05T06:09:11.758 INFO:tasks.workunit.client.0.smithi188.stderr:3+0 records out 2024-02-05T06:09:11.758 INFO:tasks.workunit.client.0.smithi188.stderr:768 bytes copied, 6.1739e-05 s, 12.4 MB/s 2024-02-05T06:09:11.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.483225 2024-02-05T06:09:11.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-02-05T06:09:11.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-02-05T06:09:12.073 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap4 2024-02-05T06:09:12.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=4 2024-02-05T06:09:12.076 INFO:tasks.workunit.client.0.smithi188.stderr:4+0 records in 2024-02-05T06:09:12.076 INFO:tasks.workunit.client.0.smithi188.stderr:4+0 records out 2024-02-05T06:09:12.076 INFO:tasks.workunit.client.0.smithi188.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.5857e-05 s, 15.5 MB/s 2024-02-05T06:09:12.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:09:12.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.483225 2024-02-05T06:09:12.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-02-05T06:09:12.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-02-05T06:09:12.492 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap5 2024-02-05T06:09:12.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-02-05T06:09:12.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-02-05T06:09:12.769 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap6 2024-02-05T06:09:12.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=6 2024-02-05T06:09:12.771 INFO:tasks.workunit.client.0.smithi188.stderr:6+0 records in 2024-02-05T06:09:12.771 INFO:tasks.workunit.client.0.smithi188.stderr:6+0 records out 2024-02-05T06:09:12.772 INFO:tasks.workunit.client.0.smithi188.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.0514e-05 s, 21.8 MB/s 2024-02-05T06:09:12.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:09:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-02-05T06:09:12.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-02-05T06:09:13.012 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap7 2024-02-05T06:09:13.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-02-05T06:09:13.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-02-05T06:09:13.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-02-05T06:09:13.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-02-05T06:09:13.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:09:13.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:09:13.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:09:13.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:09:13.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:09:13.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:09:13.687 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-02-05T06:09:15.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:15.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-02-05T06:09:15.880 INFO:tasks.workunit.client.0.smithi188.stdout:WARNING: only removing head with clones present 2024-02-05T06:09:15.880 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T06:09:16.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:09:16.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-02-05T06:09:17.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:17.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:09:17.934 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:c52c9666:::obj5:2# 2024-02-05T06:09:18.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:09:18.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-02-05T06:09:19.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:19.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:19.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:09:20.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:09:20.273 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:20.274 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:20.275 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:20.276 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:20.276 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:20.276 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:20.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:09:20.276 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:09:20.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:09:20.917 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:c52c9666:::obj5:1# 2024-02-05T06:09:21.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:09:22.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:09:22.472 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:22.472 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:09:22.472 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:22.472 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:09:22.472 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:09:22.472 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:09:22.472 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:09:22.473 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:22.474 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:22.476 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:22.476 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:22.476 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:22.476 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:09:22.476 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:22.476 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:22.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:09:22.477 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:09:22.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:09:22.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:09:22.478 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-02-05T06:09:23.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:23.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=18 2024-02-05T06:09:23.273 INFO:tasks.workunit.client.0.smithi188.stderr:18+0 records in 2024-02-05T06:09:23.273 INFO:tasks.workunit.client.0.smithi188.stderr:18+0 records out 2024-02-05T06:09:23.273 INFO:tasks.workunit.client.0.smithi188.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000146959 s, 31.4 MB/s 2024-02-05T06:09:23.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:09:24.364 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-02-05T06:09:25.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:25.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=15 2024-02-05T06:09:25.919 INFO:tasks.workunit.client.0.smithi188.stderr:15+0 records in 2024-02-05T06:09:25.919 INFO:tasks.workunit.client.0.smithi188.stderr:15+0 records out 2024-02-05T06:09:25.919 INFO:tasks.workunit.client.0.smithi188.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000107406 s, 35.8 MB/s 2024-02-05T06:09:25.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:09:27.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-02-05T06:09:27.017 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-02-05T06:09:27.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:27.953 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:09:28.539 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:0ee9ae15:::obj4:7# 2024-02-05T06:09:29.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:09:30.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:09:30.099 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:09:30.099 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:09:30.100 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:30.101 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.102 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.103 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.103 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:09:30.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-02-05T06:09:30.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-02-05T06:09:30.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:30.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:09:31.453 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:461f8b5e:::obj16:7# 2024-02-05T06:09:31.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:09:33.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:09:33.008 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:09:33.008 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:09:33.008 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:09:33.008 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:33.009 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:09:33.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:09:33.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-02-05T06:09:33.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:33.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-02-05T06:09:34.815 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:34.816 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-02-05T06:09:34.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:34.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=7 2024-02-05T06:09:34.818 INFO:tasks.workunit.client.0.smithi188.stderr:7+0 records in 2024-02-05T06:09:34.818 INFO:tasks.workunit.client.0.smithi188.stderr:7+0 records out 2024-02-05T06:09:34.818 INFO:tasks.workunit.client.0.smithi188.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.3367e-05 s, 19.2 MB/s 2024-02-05T06:09:34.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:09:36.237 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-02-05T06:09:37.032 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:37.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:37.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-02-05T06:09:38.346 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-02-05T06:09:39.101 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:39.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:39.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-02-05T06:09:40.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-02-05T06:09:41.630 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:41.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:41.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-02-05T06:09:42.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-02-05T06:09:43.583 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:44.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:44.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-02-05T06:09:45.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-02-05T06:09:46.097 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:46.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:46.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-02-05T06:09:47.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-02-05T06:09:48.108 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:48.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:48.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-02-05T06:09:49.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-02-05T06:09:50.178 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:50.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:50.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-02-05T06:09:51.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-02-05T06:09:52.123 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:52.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:52.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-02-05T06:09:53.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-02-05T06:09:54.184 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:54.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:54.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-02-05T06:09:56.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-02-05T06:09:56.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-02-05T06:09:56.764 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:09:57.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:09:57.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-02-05T06:09:58.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-02-05T06:09:58.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-02-05T06:09:58.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.483225 2024-02-05T06:09:58.067 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2024-02-05T06:09:58.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2024-02-05T06:09:58.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:09:58.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2024-02-05T06:09:58.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:09:58.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:09:58.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:09:58.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:09:58.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:09:58.075 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:09:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:09:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:09:58.076 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-02-05T06:09:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:09:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:09:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:09:58.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:09:58.099 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:09:58.126+0000 7fa70a4254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:58.101 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:09:58.129+0000 7fa70a4254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:58.102 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:09:58.130+0000 7fa70a4254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:09:58.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:09:58.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:09:58.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:09:58.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:09:58.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:09:58.348 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:09:58.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:58.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:09:58.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:58.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:58.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:09:58.619 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:09:58.647+0000 7fa70a4254c0 -1 Falling back to public interface 2024-02-05T06:09:59.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:09:59.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:09:59.614 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:09:59.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:09:59.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:09:59.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:09:59.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:00.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:00.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:00.890 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:10:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:10:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:01.013 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:01.041+0000 7fa70a4254c0 -1 osd.0 18 log_to_monitors true 2024-02-05T06:10:01.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:02.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:02.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:02.184 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:10:02.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:10:02.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:02.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:02.446 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/2268860804,v1:127.0.0.1:6803/2268860804] [v2:127.0.0.1:6804/2268860804,v1:127.0.0.1:6805/2268860804] exists,up 7575750e-c9cf-4e21-ae7c-a91d34c20027 2024-02-05T06:10:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:10:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:10:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:10:02.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2024-02-05T06:10:02.540 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:02.540 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:10:02.540 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:02.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2024-02-05T06:10:02.642 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:02.642 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:10:02.643 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:02.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max 1 2024-02-05T06:10:02.746 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:02.747 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:10:02.747 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:02.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:215: TEST_scrub_snaps: wait_for_clean 2024-02-05T06:10:02.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:10:02.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:10:02.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:10:02.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:10:02.757 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:10:02.757 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:10:02.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:10:02.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:10:02.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:10:02.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:10:02.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:10:02.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:10:02.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:10:02.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:10:02.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:10:03.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids=0 2024-02-05T06:10:03.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:10:03.104 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:10:03.105 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:10:03.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=90194313218 2024-02-05T06:10:03.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 90194313218 2024-02-05T06:10:03.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-90194313218' 2024-02-05T06:10:03.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:10:03.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-90194313218 2024-02-05T06:10:03.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:10:03.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:10:03.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-90194313218 2024-02-05T06:10:03.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:10:03.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=90194313218 2024-02-05T06:10:03.207 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 90194313218 2024-02-05T06:10:03.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 90194313218' 2024-02-05T06:10:03.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:03.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 90194313218 2024-02-05T06:10:03.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:10:04.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:10:04.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:10:04.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 90194313218 -lt 90194313218 2024-02-05T06:10:04.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:10:04.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:04.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:05.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:10:05.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:10:05.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:10:05.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:10:05.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:10:05.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:10:05.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:10:05.153 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:10:05.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:10:05.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:10:05.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:10:05.416 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:10:05.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:10:05.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:10:05.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:10:05.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:217: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2024-02-05T06:10:05.880 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:05.880 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_shallow_scrub_chunk_max": "25" 2024-02-05T06:10:05.880 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:05.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2024-02-05T06:10:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:05.985 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_shallow_scrub_chunk_min": "5" 2024-02-05T06:10:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:05.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max 2024-02-05T06:10:06.099 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:06.099 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_pg_stat_report_interval_max": "1" 2024-02-05T06:10:06.099 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:06.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2024-02-05T06:10:06.200 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:06.200 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_chunk_max": "25" 2024-02-05T06:10:06.200 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:06.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2024-02-05T06:10:06.301 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:10:06.301 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_chunk_min": "5" 2024-02-05T06:10:06.301 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:10:06.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: local pgid=1.0 2024-02-05T06:10:06.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:224: TEST_scrub_snaps: pg_scrub 1.0 2024-02-05T06:10:06.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-02-05T06:10:06.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-02-05T06:10:06.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:10:06.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:10:06.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:10:06.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:10:06.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:06.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-02-05T06:10:06.827 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0 on osd.0 to scrub 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:10:06.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:10:06.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:10:06.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:10:06.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:10:07.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:09:03.861535+0000 '>' 2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:07.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:10:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:10:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:10:08.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:10:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:10:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:10:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:10:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:10:08.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:09:03.861535+0000 '>' 2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:08.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:10:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:10:09.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:10:09.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:10:09.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:10:09.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:10:09.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:10:09.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:10:09.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:09:03.861535+0000 '>' 2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:09.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:10:10.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:10:10.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:10:10.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:10:10.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:10:10.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:10:10.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:10:10.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:10:10.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:09:03.861535+0000 '>' 2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:10.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:10:11.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:10:11.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:10:11.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:10:11.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:10:11.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:10:11.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:10:11.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:10:12.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:09:03.861535+0000 '>' 2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:12.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:10:13.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:10:13.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:10:13.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:10:13.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:10:13.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:10:13.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:10:13.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:10:13.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:10:07.080522+0000 '>' 2024-02-05T06:09:03.861535+0000 2024-02-05T06:10:13.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:10:13.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:228: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:13.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:228: TEST_scrub_snaps: wc -l 2024-02-05T06:10:13.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:228: TEST_scrub_snaps: test 2 = 2 2024-02-05T06:10:13.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: rados list-inconsistent-pg test 2024-02-05T06:10:13.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2024-02-05T06:10:13.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: test 1 = 1 2024-02-05T06:10:13.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2024-02-05T06:10:13.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1.0 = 1.0 2024-02-05T06:10:13.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2024-02-05T06:10:13.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:241: TEST_scrub_snaps: jq .inconsistents 2024-02-05T06:10:13.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:241: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:10:13.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:248: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:10:13.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:248: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:10:13.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:249: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:10:13.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:10:13.573 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2024-02-05T06:10:13.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: jq .inconsistents 2024-02-05T06:10:13.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:10:13.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:632: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:10:13.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:632: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:10:13.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:633: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:10:13.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:10:13.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: test no = yes 2024-02-05T06:10:13.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:639: TEST_scrub_snaps: test '' = yes 2024-02-05T06:10:13.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:644: TEST_scrub_snaps: find td/osd-scrub-snaps 2024-02-05T06:10:13.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:644: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:644: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:645: TEST_scrub_snaps: pids= 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: for pidfile in ${pidfiles} 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: pids+='502096 ' 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:651: TEST_scrub_snaps: ERRORS=0 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: seq 1 7 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:10:13.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap1 2024-02-05T06:10:13.841 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap1 2024-02-05T06:10:13.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:10:13.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap2 2024-02-05T06:10:14.159 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap2 2024-02-05T06:10:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:10:14.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap3 2024-02-05T06:10:14.343 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap3 2024-02-05T06:10:14.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:10:14.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap4 2024-02-05T06:10:14.574 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap4 2024-02-05T06:10:14.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:10:14.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap5 2024-02-05T06:10:14.829 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap5 2024-02-05T06:10:14.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:10:14.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap6 2024-02-05T06:10:15.030 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap6 2024-02-05T06:10:15.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: for i in `seq 1 7` 2024-02-05T06:10:15.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: rados -p test rmsnap snap7 2024-02-05T06:10:15.223 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap7 2024-02-05T06:10:15.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: sleep 5 2024-02-05T06:10:20.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:658: TEST_scrub_snaps: local -i loop=0 2024-02-05T06:10:20.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: ceph pg dump pgs 2024-02-05T06:10:20.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: grep -q snaptrim 2024-02-05T06:10:20.485 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:10:20.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2024-02-05T06:10:20.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim_error 2024-02-05T06:10:20.763 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:10:20.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: break 2024-02-05T06:10:20.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:672: TEST_scrub_snaps: ceph pg dump pgs 2024-02-05T06:10:21.022 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:10:21.022 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2024-02-05T06:10:15.277535+0000 23'60 29:123 [0] 0 [0] 0 18'56 2024-02-05T06:10:07.080522+0000 0'0 2024-02-05T06:09:03.861535+0000 7 1 periodic scrub scheduled @ 2024-02-06T18:03:12.479472+0000 33 0 2024-02-05T06:10:21.022 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T06:10:21.022 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:10:21.023 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:10:21.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: for pid in $pids 2024-02-05T06:10:21.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: kill -0 502096 2024-02-05T06:10:21.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:683: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2024-02-05T06:10:21.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:21.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:21.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:21.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:21.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: declare -a err_strings 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:686: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2024-02-05T06:10:26.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2024-02-05T06:10:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-02-05T06:10:26.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2024-02-05T06:10:26.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-02-05T06:10:26.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2024-02-05T06:10:26.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:10:26.397 INFO:tasks.workunit.client.0.smithi188.stdout:TEST PASSED 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:719: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:725: TEST_scrub_snaps: echo 'TEST PASSED' 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:726: TEST_scrub_snaps: return 0 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:26.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:26.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:26.399 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:26.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:26.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:10:26.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:10:26.404 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:10:26.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:10:26.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:10:26.405 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:10:26.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:10:26.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:10:26.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:10:26.406 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:10:26.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:10:26.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:10:26.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:10:26.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:10:26.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:26.418 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:26.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.483225 2024-02-05T06:10:26.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:10:26.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:10:26.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-02-05T06:10:26.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-02-05T06:10:26.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-02-05T06:10:26.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-02-05T06:10:26.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:10:26.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:10:26.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:10:26.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:10:26.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:10:26.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:10:26.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:10:26.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:10:26.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:10:26.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:10:26.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:10:26.426 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:10:26.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:10:26.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:10:26.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:10:26.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:10:26.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:26.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:26.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.483225 2024-02-05T06:10:26.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:10:26.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:10:26.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-02-05T06:10:26.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:10:26.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:26.431 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:26.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.483225 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1174: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1175: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:10:26.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1176: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:731: _scrub_snaps_multi: local poolname=test 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local OBJS=16 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local OSDS=2 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local which=primary 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: TESTDATA=testdata.483225 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-02-05T06:10:26.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-02-05T06:10:26.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:10:26.792 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:10:26.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:10:26.793 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:10:26.793 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:26.793 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:26.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:10:26.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:10:26.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:10:26.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.483225/ceph-mon.a.asok 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:10:26.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.483225/ceph-mon.a.asok config get fsid 2024-02-05T06:10:27.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:10:27.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:10:27.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.483225/ceph-mon.a.asok 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:10:27.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.483225/ceph-mon.a.asok config get mon_host 2024-02-05T06:10:27.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:739: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-02-05T06:10:27.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-02-05T06:10:27.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:10:27.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:10:27.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:10:27.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-02-05T06:10:27.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:10:27.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:10:27.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:10:27.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:10:27.971 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:10:27.971 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:27.971 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:27.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:10:27.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:10:27.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:10:28.004 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:10:28.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: seq 0 1 2024-02-05T06:10:28.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:10:28.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-02-05T06:10:28.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:10:28.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:10:28.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:10:28.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2969980-9dd2-491e-9ad9-137224fb71c1 2024-02-05T06:10:28.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e2969980-9dd2-491e-9ad9-137224fb71c1' 2024-02-05T06:10:28.012 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 e2969980-9dd2-491e-9ad9-137224fb71c1 2024-02-05T06:10:28.012 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:10:28.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUe8BlJzWcAxAA8hIH6ogg+VepJ/dXd1C3PA== 2024-02-05T06:10:28.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUe8BlJzWcAxAA8hIH6ogg+VepJ/dXd1C3PA=="}' 2024-02-05T06:10:28.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2969980-9dd2-491e-9ad9-137224fb71c1 -i td/osd-scrub-snaps/0/new.json 2024-02-05T06:10:28.310 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUe8BlJzWcAxAA8hIH6ogg+VepJ/dXd1C3PA== --osd-uuid e2969980-9dd2-491e-9ad9-137224fb71c1 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:28.384+0000 7f0551b944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:28.389+0000 7f0551b944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:28.390+0000 7f0551b944c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:28.390+0000 7f0551b944c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:28.390+0000 7f0551b944c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:10:28.400 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:28.390+0000 7f0551b944c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:10:28.445 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:28.473+0000 7f0551b944c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-02-05T06:10:33.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-02-05T06:10:33.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:10:33.896 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:10:33.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:10:33.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:10:34.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:10:34.251 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:10:34.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:10:34.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:10:34.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:10:34.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:10:34.285 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:34.313+0000 7f994419b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:34.287 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:34.315+0000 7f994419b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:34.288 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:34.316+0000 7f994419b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:10:34.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:10:34.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:10:34.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:10:34.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:10:34.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:10:34.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:34.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:10:34.545 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:10:34.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:34.545 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:34.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:34.824 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:34.853+0000 7f994419b4c0 -1 Falling back to public interface 2024-02-05T06:10:35.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:35.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:35.819 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:10:35.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:10:35.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:35.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:36.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:37.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:37.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:37.107 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:10:37.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:10:37.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:37.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:37.229 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:10:37.257+0000 7f994419b4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:10:37.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:38.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:38.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:38.397 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:10:38.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:10:38.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:38.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:38.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:39.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:39.686 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:10:39.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:39.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:10:39.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:39.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:39.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:40.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:40.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:40.976 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:10:40.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:10:40.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:40.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:41.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:42.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:42.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:42.262 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:10:42.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:10:42.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:42.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:42.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:43.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:43.550 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:10:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:10:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:43.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:43.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:44.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:44.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:44.839 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:10:44.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:10:44.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:44.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:45.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:46.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:46.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:46.144 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:10:46.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:10:46.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:46.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:46.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:47.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:47.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:47.433 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:10:47.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:10:47.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:47.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:47.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:48.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:48.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:48.806 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:10:48.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:10:48.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:48.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:49.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:50.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:50.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:50.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:10:50.095 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:10:50.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:50.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:50.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:51.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:51.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:51.461 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:10:51.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:10:51.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:51.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:52.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:53.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:53.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:53.006 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:10:53.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:10:53.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:53.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:53.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:54.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:54.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:54.651 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:10:54.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:10:54.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:54.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:55.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:56.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:56.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:56.583 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:10:56.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:10:56.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:56.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:57.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:10:58.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:10:58.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:10:58.310 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:10:58.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:10:58.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:10:58.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:10:59.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:00.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:00.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:00.084 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:11:00.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:11:00.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:00.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:11:00.464 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2392219812,v1:127.0.0.1:6803/2392219812] [v2:127.0.0.1:6804/2392219812,v1:127.0.0.1:6805/2392219812] exists,up e2969980-9dd2-491e-9ad9-137224fb71c1 2024-02-05T06:11:00.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:11:00.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:11:00.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:11:00.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:11:00.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:11:00.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:11:00.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:11:00.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:11:00.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:11:00.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:11:00.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:11:00.471 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:11:00.471 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:11:00.471 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:11:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:11:00.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:11:00.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d50b7161-da0e-4c86-ba8c-73adecc4a93b 2024-02-05T06:11:00.473 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 d50b7161-da0e-4c86-ba8c-73adecc4a93b 2024-02-05T06:11:00.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d50b7161-da0e-4c86-ba8c-73adecc4a93b' 2024-02-05T06:11:00.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:11:00.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0e8BlA/n3HhAAwQ/DWwXb+JUqegiE+JJuQw== 2024-02-05T06:11:00.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0e8BlA/n3HhAAwQ/DWwXb+JUqegiE+JJuQw=="}' 2024-02-05T06:11:00.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d50b7161-da0e-4c86-ba8c-73adecc4a93b -i td/osd-scrub-snaps/1/new.json 2024-02-05T06:11:00.821 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:11:00.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-02-05T06:11:00.832 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0e8BlA/n3HhAAwQ/DWwXb+JUqegiE+JJuQw== --osd-uuid d50b7161-da0e-4c86-ba8c-73adecc4a93b 2024-02-05T06:11:00.869 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:00.897+0000 7f4583d484c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:00.905 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:00.933+0000 7f4583d484c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:00.906 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:00.934+0000 7f4583d484c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:00.906 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:00.935+0000 7f4583d484c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:11:00.907 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:00.935+0000 7f4583d484c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:11:00.907 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:00.935+0000 7f4583d484c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:11:00.972 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:01.000+0000 7f4583d484c0 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-02-05T06:11:06.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-02-05T06:11:06.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:11:06.815 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:11:06.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:11:06.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:11:07.167 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:11:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:11:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:11:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:11:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:11:07.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:11:07.202 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:07.230+0000 7f84204184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:07.204 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:07.232+0000 7f84204184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:07.205 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:07.234+0000 7f84204184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:11:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:11:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:11:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:11:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:11:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:11:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:07.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:11:07.444 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:11:07.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:07.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:07.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:07.731 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:07.760+0000 7f84204184c0 -1 Falling back to public interface 2024-02-05T06:11:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:08.721 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:11:08.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:08.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:11:08.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:08.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:09.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:10.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:10.002 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:11:10.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:10.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:11:10.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:10.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:10.089 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:11:10.117+0000 7f84204184c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:11:10.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:11.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:11.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:11.366 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:11:11.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:11:11.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:11.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:11.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:12.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:12.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:12.652 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:11:12.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:11:12.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:12.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:12.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:13.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:13.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:13.987 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:11:13.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:11:13.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:13.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:14.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:15.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:15.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:15.274 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:11:15.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:11:15.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:15.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:15.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:16.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:16.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:16.562 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:11:16.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:11:16.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:16.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:17.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:17.939 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:11:17.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:11:17.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:17.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:18.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:19.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:19.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:19.278 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:11:19.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:11:19.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:19.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:19.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:20.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:20.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:20.564 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:11:20.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:11:20.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:20.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:20.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:21.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:21.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:21.954 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:11:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:11:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:22.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:23.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:23.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:23.419 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:11:23.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:11:23.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:23.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:24.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:25.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:25.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:25.063 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:11:25.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:11:25.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:25.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:25.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:26.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:26.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:26.500 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:11:26.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:11:26.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:26.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:26.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:27.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:27.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:27.933 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:11:27.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:11:27.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:27.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:28.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:29.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:29.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:29.967 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:11:29.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:11:29.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:29.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:30.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:31.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:31.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:31.322 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:11:31.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:11:31.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:31.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:31.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:32.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:32.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:32.820 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:11:32.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:11:32.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:32.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:33.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:11:34.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:11:34.144 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:11:34.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:11:34.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:11:34.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:11:34.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:11:34.429 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1973359265,v1:127.0.0.1:6811/1973359265] [v2:127.0.0.1:6812/1973359265,v1:127.0.0.1:6813/1973359265] exists,up d50b7161-da0e-4c86-ba8c-73adecc4a93b 2024-02-05T06:11:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:11:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:11:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:11:34.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:746: _scrub_snaps_multi: ceph osd set noscrub 2024-02-05T06:11:34.803 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:11:34.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-02-05T06:11:35.238 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:11:35.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: create_pool test 1 1 2024-02-05T06:11:35.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:11:35.615 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T06:11:35.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:11:36.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:11:36.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:11:36.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:11:36.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:11:36.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:11:36.629 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:11:36.629 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:11:36.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:11:36.629 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:11:36.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:11:36.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:11:36.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:11:36.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:11:36.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:11:36.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:11:36.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:11:37.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:11:37.001 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T06:11:37.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:11:37.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:37.002 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:11:37.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-02-05T06:11:37.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-02-05T06:11:37.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-02-05T06:11:37.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:11:37.114 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:11:37.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-02-05T06:11:37.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-02-05T06:11:37.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672966' 2024-02-05T06:11:37.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:37.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-02-05T06:11:37.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:37.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:11:37.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-02-05T06:11:37.229 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:37.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-02-05T06:11:37.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-02-05T06:11:37.230 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836492 2024-02-05T06:11:37.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:37.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-02-05T06:11:37.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:11:38.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:11:38.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:11:38.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-02-05T06:11:38.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:11:38.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-02-05T06:11:38.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:11:38.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:11:38.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-02-05T06:11:38.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:11:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-02-05T06:11:38.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-02-05T06:11:38.795 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672966 2024-02-05T06:11:38.795 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:11:39.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-02-05T06:11:39.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:11:39.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:39.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:39.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:11:39.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:11:39.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:11:39.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:11:39.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:11:39.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:11:39.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:11:39.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:11:39.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:11:39.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:11:39.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:11:39.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:11:40.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:11:40.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:11:40.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:11:40.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: ceph osd dump 2024-02-05T06:11:40.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-02-05T06:11:40.112 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: awk '{ print $2 }' 2024-02-05T06:11:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: poolid=1 2024-02-05T06:11:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.483225 bs=1032 count=1 2024-02-05T06:11:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:11:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:11:40.388 INFO:tasks.workunit.client.0.smithi188.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 6.273e-05 s, 16.5 MB/s 2024-02-05T06:11:40.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: seq 1 16 2024-02-05T06:11:40.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:40.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj1 testdata.483225 2024-02-05T06:11:40.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:40.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj2 testdata.483225 2024-02-05T06:11:40.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:40.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj3 testdata.483225 2024-02-05T06:11:40.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:40.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj4 testdata.483225 2024-02-05T06:11:40.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:40.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj5 testdata.483225 2024-02-05T06:11:41.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:41.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj6 testdata.483225 2024-02-05T06:11:41.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:41.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj7 testdata.483225 2024-02-05T06:11:41.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:41.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj8 testdata.483225 2024-02-05T06:11:41.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:41.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj9 testdata.483225 2024-02-05T06:11:41.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:41.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj10 testdata.483225 2024-02-05T06:11:41.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:41.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj11 testdata.483225 2024-02-05T06:11:41.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:41.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj12 testdata.483225 2024-02-05T06:11:42.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:42.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj13 testdata.483225 2024-02-05T06:11:42.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:42.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj14 testdata.483225 2024-02-05T06:11:42.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:42.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj15 testdata.483225 2024-02-05T06:11:42.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:11:42.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj16 testdata.483225 2024-02-05T06:11:42.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: get_primary test obj1 2024-02-05T06:11:42.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:11:42.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:11:42.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:11:42.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: local primary=1 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: get_not_primary test obj1 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:11:42.842 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:11:43.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:11:43.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-02-05T06:11:43.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:11:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: local replica=0 2024-02-05T06:11:43.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.483225 '$primary' 2024-02-05T06:11:43.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.483225 1 2024-02-05T06:11:43.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-02-05T06:11:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-02-05T06:11:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.483225 2024-02-05T06:11:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2024-02-05T06:11:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-02-05T06:11:43.396 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-02-05T06:11:43.665 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap1 2024-02-05T06:11:43.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=1 2024-02-05T06:11:43.668 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:11:43.668 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:11:43.668 INFO:tasks.workunit.client.0.smithi188.stderr:256 bytes copied, 5.717e-05 s, 4.5 MB/s 2024-02-05T06:11:43.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.483225 2024-02-05T06:11:43.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:11:44.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.483225 2024-02-05T06:11:44.205 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-02-05T06:11:44.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:44.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.483225 2024-02-05T06:11:44.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:44.356 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.483225 2024-02-05T06:11:44.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:44.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.483225 2024-02-05T06:11:44.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:44.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.483225 2024-02-05T06:11:44.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:44.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.483225 2024-02-05T06:11:44.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:44.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.483225 2024-02-05T06:11:44.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:44.984 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.483225 2024-02-05T06:11:45.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:45.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.483225 2024-02-05T06:11:45.243 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:11:45.244 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.483225 2024-02-05T06:11:45.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-02-05T06:11:45.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-02-05T06:11:45.543 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap2 2024-02-05T06:11:45.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=2 2024-02-05T06:11:45.546 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records in 2024-02-05T06:11:45.546 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records out 2024-02-05T06:11:45.547 INFO:tasks.workunit.client.0.smithi188.stderr:512 bytes copied, 6.1751e-05 s, 8.3 MB/s 2024-02-05T06:11:45.547 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:11:45.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-02-05T06:11:45.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-02-05T06:11:45.937 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap3 2024-02-05T06:11:45.939 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=3 2024-02-05T06:11:45.940 INFO:tasks.workunit.client.0.smithi188.stderr:3+0 records in 2024-02-05T06:11:45.940 INFO:tasks.workunit.client.0.smithi188.stderr:3+0 records out 2024-02-05T06:11:45.940 INFO:tasks.workunit.client.0.smithi188.stderr:768 bytes copied, 6.4685e-05 s, 11.9 MB/s 2024-02-05T06:11:45.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.483225 2024-02-05T06:11:46.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-02-05T06:11:46.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-02-05T06:11:46.407 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap4 2024-02-05T06:11:46.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=4 2024-02-05T06:11:46.409 INFO:tasks.workunit.client.0.smithi188.stderr:4+0 records in 2024-02-05T06:11:46.409 INFO:tasks.workunit.client.0.smithi188.stderr:4+0 records out 2024-02-05T06:11:46.409 INFO:tasks.workunit.client.0.smithi188.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.4091e-05 s, 16.0 MB/s 2024-02-05T06:11:46.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:11:46.651 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.483225 2024-02-05T06:11:46.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-02-05T06:11:46.802 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-02-05T06:11:46.960 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap5 2024-02-05T06:11:46.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-02-05T06:11:46.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-02-05T06:11:47.202 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap6 2024-02-05T06:11:47.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=6 2024-02-05T06:11:47.205 INFO:tasks.workunit.client.0.smithi188.stderr:6+0 records in 2024-02-05T06:11:47.205 INFO:tasks.workunit.client.0.smithi188.stderr:6+0 records out 2024-02-05T06:11:47.205 INFO:tasks.workunit.client.0.smithi188.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.0888e-05 s, 21.7 MB/s 2024-02-05T06:11:47.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:11:47.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-02-05T06:11:47.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-02-05T06:11:47.638 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap7 2024-02-05T06:11:47.640 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-02-05T06:11:47.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-02-05T06:11:48.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-02-05T06:11:48.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-02-05T06:11:48.117 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:11:48.117 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:11:48.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:11:48.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:11:48.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:11:48.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:11:48.423 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2024-02-05T06:11:49.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:11:49.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-02-05T06:11:50.557 INFO:tasks.workunit.client.0.smithi188.stdout:WARNING: only removing head with clones present 2024-02-05T06:11:50.557 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T06:11:51.073 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-02-05T06:11:51.073 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-02-05T06:11:51.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:11:51.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:11:52.502 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:c52c9666:::obj5:2# 2024-02-05T06:11:53.021 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-02-05T06:11:53.021 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-02-05T06:11:54.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:11:54.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:11:54.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:11:54.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:54.867 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:54.868 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:54.869 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:11:54.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:11:54.870 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:11:54.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:11:55.415 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:c52c9666:::obj5:1# 2024-02-05T06:11:55.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:11:56.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:11:56.977 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:56.977 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:11:56.977 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:56.978 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:11:56.979 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:11:56.980 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:11:56.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:11:56.981 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:11:56.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:11:56.981 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2024-02-05T06:11:56.981 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-02-05T06:11:57.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:11:57.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=18 2024-02-05T06:11:57.772 INFO:tasks.workunit.client.0.smithi188.stderr:18+0 records in 2024-02-05T06:11:57.772 INFO:tasks.workunit.client.0.smithi188.stderr:18+0 records out 2024-02-05T06:11:57.772 INFO:tasks.workunit.client.0.smithi188.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.0001205 s, 38.2 MB/s 2024-02-05T06:11:57.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:11:58.931 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2024-02-05T06:12:00.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:00.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=15 2024-02-05T06:12:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:15+0 records in 2024-02-05T06:12:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:15+0 records out 2024-02-05T06:12:00.415 INFO:tasks.workunit.client.0.smithi188.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000107032 s, 35.9 MB/s 2024-02-05T06:12:00.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:12:01.491 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2024-02-05T06:12:01.491 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-02-05T06:12:02.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:02.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:12:03.096 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:0ee9ae15:::obj4:7# 2024-02-05T06:12:03.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:12:04.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:12:04.658 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:12:04.659 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:04.660 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.661 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:12:04.661 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:04.661 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.661 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:12:04.661 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:12:04.661 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.662 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:12:04.662 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:12:04.662 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.662 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:12:04.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:12:04.663 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2024-02-05T06:12:04.664 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-02-05T06:12:05.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:05.435 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:12:05.933 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:461f8b5e:::obj16:7# 2024-02-05T06:12:06.448 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2024-02-05T06:12:07.494 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:12:07.495 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:12:07.496 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:12:07.497 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:12:07.497 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:12:07.497 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:12:07.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:12:07.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:12:07.498 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2024-02-05T06:12:08.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:08.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-02-05T06:12:09.430 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:09.430 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-02-05T06:12:09.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:09.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=7 2024-02-05T06:12:09.433 INFO:tasks.workunit.client.0.smithi188.stderr:7+0 records in 2024-02-05T06:12:09.433 INFO:tasks.workunit.client.0.smithi188.stderr:7+0 records out 2024-02-05T06:12:09.433 INFO:tasks.workunit.client.0.smithi188.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.3421e-05 s, 19.2 MB/s 2024-02-05T06:12:09.433 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:12:10.856 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2024-02-05T06:12:11.518 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:11.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:11.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-02-05T06:12:12.944 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2024-02-05T06:12:13.622 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:14.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:14.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-02-05T06:12:15.449 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2024-02-05T06:12:16.176 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:16.447 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:16.447 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-02-05T06:12:17.487 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2024-02-05T06:12:18.271 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:19.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:19.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-02-05T06:12:20.081 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2024-02-05T06:12:20.726 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:20.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:20.991 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-02-05T06:12:22.036 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2024-02-05T06:12:22.704 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:22.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:22.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-02-05T06:12:24.011 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2024-02-05T06:12:24.691 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:24.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:24.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-02-05T06:12:25.974 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2024-02-05T06:12:26.610 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:26.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:26.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-02-05T06:12:27.917 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2024-02-05T06:12:28.599 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:29.119 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:29.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-02-05T06:12:30.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-02-05T06:12:30.210 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2024-02-05T06:12:30.860 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:12:31.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:12:31.127 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-02-05T06:12:32.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-02-05T06:12:32.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-02-05T06:12:32.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: rm -f testdata.483225 2024-02-05T06:12:32.177 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:12:32.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: seq 0 1 2024-02-05T06:12:32.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:12:32.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:12:32.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:12:32.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:12:32.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:12:32.186 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:12:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:12:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:12:32.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-02-05T06:12:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:12:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:12:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:12:32.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:12:32.220 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:32.249+0000 7f666032b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:32.223 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:32.252+0000 7f666032b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:32.224 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:32.253+0000 7f666032b4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:32.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:12:32.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:12:32.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:12:32.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:12:32.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:12:32.468 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:12:32.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:32.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:12:32.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:32.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:32.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:32.756 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:32.785+0000 7f666032b4c0 -1 Falling back to public interface 2024-02-05T06:12:33.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:33.744 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:12:33.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:33.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:12:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:33.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:34.048 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:35.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:35.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:35.049 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:12:35.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:12:35.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:35.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:35.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:35.493 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:35.522+0000 7f666032b4c0 -1 osd.0 23 log_to_monitors true 2024-02-05T06:12:36.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:36.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:36.335 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:12:36.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:12:36.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:36.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:12:36.609 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/1923070130,v1:127.0.0.1:6803/1923070130] [v2:127.0.0.1:6804/1923070130,v1:127.0.0.1:6805/1923070130] exists,up e2969980-9dd2-491e-9ad9-137224fb71c1 2024-02-05T06:12:36.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:12:36.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:12:36.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:12:36.614 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:12:36.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:12:36.616 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:12:36.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:12:36.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-02-05T06:12:36.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:12:36.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:12:36.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:12:36.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:12:36.646 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:36.675+0000 7f66368494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:36.677+0000 7f66368494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:36.649 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:36.678+0000 7f66368494c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:12:36.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:12:36.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:12:36.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:12:36.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:12:36.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:12:36.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:36.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:12:36.933 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:12:36.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:36.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:37.164 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:37.193+0000 7f66368494c0 -1 Falling back to public interface 2024-02-05T06:12:37.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:38.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:38.209 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:12:38.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:12:38.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:38.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:38.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:12:39.415 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:12:39.444+0000 7f66368494c0 -1 osd.1 23 log_to_monitors true 2024-02-05T06:12:39.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:12:39.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:12:39.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:12:39.494 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:12:39.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:12:39.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:12:39.784 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 29 up_thru 14 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/485447313,v1:127.0.0.1:6811/485447313] [v2:127.0.0.1:6812/485447313,v1:127.0.0.1:6813/485447313] exists,up d50b7161-da0e-4c86-ba8c-73adecc4a93b 2024-02-05T06:12:39.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:12:39.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:12:39.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:12:39.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:772: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-02-05T06:12:39.941 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:12:39.941 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:12:39.941 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:39.955 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:12:39.955 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:12:39.955 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:39.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:773: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-02-05T06:12:40.069 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:12:40.069 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:12:40.069 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:40.083 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:12:40.083 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:12:40.083 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:40.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-02-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:40.214 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:12:40.214 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:12:40.214 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:40.223 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max 1 2024-02-05T06:12:40.351 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:12:40.351 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:12:40.351 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:40.365 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:12:40.365 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:12:40.365 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:40.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:12:40.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:12:40.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:12:40.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:12:40.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:12:40.375 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:12:40.375 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:12:40.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:12:40.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:12:40.376 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:12:40.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:12:40.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:12:40.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:12:40.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:12:40.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:12:40.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:12:40.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:12:40.790 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T06:12:40.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:12:40.790 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:40.790 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:12:40.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-02-05T06:12:40.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-02-05T06:12:40.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699' 2024-02-05T06:12:40.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:12:40.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:12:41.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T06:12:41.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T06:12:41.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-02-05T06:12:41.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:41.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149699 2024-02-05T06:12:41.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:41.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:12:41.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149699 2024-02-05T06:12:41.037 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:41.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-02-05T06:12:41.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-02-05T06:12:41.038 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 111669149699 2024-02-05T06:12:41.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:41.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149699 2024-02-05T06:12:41.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:12:42.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:12:42.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:12:42.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-02-05T06:12:42.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:12:42.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-02-05T06:12:42.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:12:42.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:12:42.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-02-05T06:12:42.595 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:12:42.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T06:12:42.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-02-05T06:12:42.597 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 124554051586 2024-02-05T06:12:42.597 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:12:42.915 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T06:12:42.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:12:42.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:42.915 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:43.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:12:43.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:12:43.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:12:43.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:12:43.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:12:43.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:12:43.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:12:43.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:12:43.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:12:43.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:12:43.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:12:43.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: local pgid=1.0 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: pg_scrub 1.0 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:43.906 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:43.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:12:44.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:44.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-02-05T06:12:44.436 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0 on osd.1 to scrub 2024-02-05T06:12:44.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:44.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:12:44.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:44.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:12:44.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:11:35.599023+0000 '>' 2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:44.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:45.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:45.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:45.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:12:45.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:12:45.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:45.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:45.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:12:46.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:11:35.599023+0000 '>' 2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:46.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:47.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:47.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:47.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:12:47.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:12:47.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:47.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:47.011 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:12:47.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:11:35.599023+0000 '>' 2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:47.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:48.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:48.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:48.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:12:48.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:12:48.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:48.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:48.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:12:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:11:35.599023+0000 '>' 2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:48.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:49.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:49.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:49.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:12:49.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:12:49.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:49.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:49.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:12:49.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:11:35.599023+0000 '>' 2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:49.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:12:50.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:12:50.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:12:50.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:12:50.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:12:50.869 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:12:50.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:12:50.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:12:51.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:12:45.869360+0000 '>' 2024-02-05T06:11:35.599023+0000 2024-02-05T06:12:51.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:12:51.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:12:51.150 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: wc -l 2024-02-05T06:12:51.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: test 14 -gt 3 2024-02-05T06:12:51.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:12:51.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: wc -l 2024-02-05T06:12:51.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: test 14 -gt 3 2024-02-05T06:12:51.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-02-05T06:12:51.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-02-05T06:12:51.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: test 1 = 1 2024-02-05T06:12:51.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-02-05T06:12:51.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: test 1.0 = 1.0 2024-02-05T06:12:51.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:792: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "epoch": 29, 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "inconsistents": [ 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 5 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.227 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 1, 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.228 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'19", 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'5", 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4243.0:1", 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 5, 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:40.994446+0000", 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:40.995008+0000", 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.229 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x46721689", 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.230 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.231 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.232 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 2, 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 20 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 2, 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:12:51.233 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "18'41", 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "17'20", 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4291.0:1", 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 20, 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:43.986165+0000", 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:43.986727+0000", 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.234 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.235 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.236 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.237 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 4, 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 42 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch" 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.238 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 4, 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "20'45", 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "18'42", 2024-02-05T06:12:51.239 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4327.0:1", 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 42, 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 512, 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:45.608148+0000", 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:45.622401+0000", 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xe9d177b6", 2024-02-05T06:12:51.240 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.241 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 512 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 4608, 2024-02-05T06:12:51.242 INFO:tasks.workunit.client.0.smithi188.stdout: "object_info": { 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 4, 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "20'45", 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "18'42", 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4327.0:1", 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 42, 2024-02-05T06:12:51.243 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 512, 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:45.608148+0000", 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:45.622401+0000", 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xe9d177b6", 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.244 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj4", 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.245 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 7, 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 4 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj4", 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 7, 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2826278768, 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.246 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "23'51", 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'4", 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4240.0:1", 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 4, 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:40.868023+0000", 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:40.868573+0000", 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.247 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x46721689", 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.248 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.249 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 7, 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 0 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "missing", 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: "info_missing" 2024-02-05T06:12:51.250 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.251 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "info_missing" 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1792 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj1", 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.252 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 18 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj1", 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1828249343, 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.253 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'18", 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'1", 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4288.0:1", 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 18, 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:43.730932+0000", 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:43.821291+0000", 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.254 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.255 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.256 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj10", 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 32 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.257 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj10", 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 718195851, 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'32", 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'10", 2024-02-05T06:12:51.258 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4309.0:1", 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 32, 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:44.794159+0000", 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:44.794795+0000", 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.259 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.260 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.261 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "????", 2024-02-05T06:12:51.262 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj11", 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.263 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 34 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj11", 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 693400951, 2024-02-05T06:12:51.264 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'34", 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'11", 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4312.0:1", 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 34, 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:44.919803+0000", 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:44.920349+0000", 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.265 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.266 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.267 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.268 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.269 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj13", 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 38 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.270 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj13", 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2087409765, 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'38", 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'13", 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4318.0:1", 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 38, 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:45.171175+0000", 2024-02-05T06:12:51.271 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:45.171733+0000", 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.272 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.273 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.274 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]" 2024-02-05T06:12:51.275 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj14", 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.276 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 40 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj14", 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2484217095, 2024-02-05T06:12:51.277 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'40", 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'14", 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4321.0:1", 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 40, 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:45.305433+0000", 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:45.306004+0000", 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.278 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.279 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.280 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.281 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1033, 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.282 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj15", 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 15 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.283 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_corrupted" 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj15", 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 612772309, 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.284 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "16'15", 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "0'0", 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4273.0:1", 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 15, 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:42.376367+0000", 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:42.376914+0000", 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.285 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.286 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.286 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.286 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.286 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x46721689", 2024-02-05T06:12:51.286 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.286 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.286 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.287 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.287 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.287 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.287 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.287 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:12:51.288 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_corrupted" 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": "Z2FyYmFnZQo=" 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.289 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj2", 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 56 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_missing" 2024-02-05T06:12:51.290 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj2", 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1058988552, 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "23'56", 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "20'48", 2024-02-05T06:12:51.291 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4363.0:1", 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 56, 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 0, 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:48.065730+0000", 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:48.066246+0000", 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "whiteout", 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty" 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xffffffff", 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.292 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 0, 2024-02-05T06:12:51.293 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 7, 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 4, 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: 4, 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: 3, 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: 2, 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.294 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 7, 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1024, 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: 7, 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: 6, 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: 5 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.295 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_missing" 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 0 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj3", 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.296 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 44 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch" 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj3", 2024-02-05T06:12:51.297 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1643547569, 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "19'44", 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "17'22", 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4333.0:1", 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 44, 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 768, 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:46.003089+0000", 2024-02-05T06:12:51.298 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:46.024812+0000", 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x7ee53007", 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.299 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 768 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.300 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 3840, 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "object_info": { 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj3", 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1643547569, 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.301 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "19'44", 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "17'22", 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4333.0:1", 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 44, 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 768, 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:46.003089+0000", 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:46.024812+0000", 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.302 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x7ee53007", 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.303 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj6", 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 24 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.304 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj6", 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2202164420, 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'24", 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'6", 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4297.0:1", 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 24, 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.305 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:44.279673+0000", 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:44.280228+0000", 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.306 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.307 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.308 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj7", 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.309 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 26 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj7", 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1552453721, 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.310 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'26", 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'7", 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4300.0:1", 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 26, 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:44.417570+0000", 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:44.418143+0000", 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.311 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.312 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.313 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.314 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj8", 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 28 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.315 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj8", 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2381834917, 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'28", 2024-02-05T06:12:51.316 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'8", 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4303.0:1", 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 28, 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:44.543611+0000", 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:44.544211+0000", 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.317 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.318 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.319 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.320 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.321 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj9", 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 30 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:12:51.322 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj9", 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 3833113727, 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'30", 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'9", 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4306.0:1", 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 30, 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.323 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:11:44.668775+0000", 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:11:44.669346+0000", 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xc0b13746", 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:12:51.324 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:12:51.325 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.326 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "size": "????", 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.327 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.328 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.328 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.328 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.328 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.328 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:12:51.328 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:12:51.328 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:12:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:794: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-02-05T06:12:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: '[' primary = replica ']' 2024-02-05T06:12:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:808: _scrub_snaps_multi: scruberrors=30 2024-02-05T06:12:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:809: _scrub_snaps_multi: jq .inconsistents 2024-02-05T06:12:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:809: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:12:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:12:51.331 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1064: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: test no = yes 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1070: _scrub_snaps_multi: test '' = yes 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: pids= 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:12:51.332 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='524902 ' 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='524338 ' 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: ERRORS=0 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1086: _scrub_snaps_multi: '[' primary = primary ']' 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: seq 1 7 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:12:51.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap1 2024-02-05T06:12:51.475 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap1 2024-02-05T06:12:51.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:12:51.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap2 2024-02-05T06:12:51.726 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap2 2024-02-05T06:12:51.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:12:51.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap3 2024-02-05T06:12:51.969 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap3 2024-02-05T06:12:51.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:12:51.971 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap4 2024-02-05T06:12:52.220 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap4 2024-02-05T06:12:52.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:12:52.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap5 2024-02-05T06:12:52.487 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap5 2024-02-05T06:12:52.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:12:52.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap6 2024-02-05T06:12:52.688 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap6 2024-02-05T06:12:52.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1088: _scrub_snaps_multi: for i in `seq 1 7` 2024-02-05T06:12:52.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1090: _scrub_snaps_multi: rados -p test rmsnap snap7 2024-02-05T06:12:52.889 INFO:tasks.workunit.client.0.smithi188.stdout:removed pool test snap snap7 2024-02-05T06:12:52.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1092: _scrub_snaps_multi: sleep 5 2024-02-05T06:12:57.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: local -i loop=0 2024-02-05T06:12:57.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1094: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:12:57.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1094: _scrub_snaps_multi: grep -q snaptrim 2024-02-05T06:12:58.163 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:12:58.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:12:58.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: grep -q snaptrim_error 2024-02-05T06:12:58.435 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:12:58.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1098: _scrub_snaps_multi: break 2024-02-05T06:12:58.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1108: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:12:58.704 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:12:58.704 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2024-02-05T06:12:52.955335+0000 23'56 37:119 [1,0] 1 [1,0] 1 23'56 2024-02-05T06:12:45.869360+0000 0'0 2024-02-05T06:11:35.599023+0000 7 1 periodic scrub scheduled @ 2024-02-06T08:42:00.876736+0000 33 0 2024-02-05T06:12:58.704 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T06:12:58.704 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:12:58.704 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:12:58.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:12:58.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 524902 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 524338 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:12:58.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:13:04.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1121: _scrub_snaps_multi: declare -a err_strings 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1123: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:13:04.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1141: _scrub_snaps_multi: declare -a rep_err_strings 2024-02-05T06:13:04.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: eval echo '$primary' 2024-02-05T06:13:04.050 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: echo 1 2024-02-05T06:13:04.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: osd=1 2024-02-05T06:13:04.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1143: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-02-05T06:13:04.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-02-05T06:13:04.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:13:04.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1153: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-02-05T06:13:04.052 INFO:tasks.workunit.client.0.smithi188.stdout:TEST PASSED 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1159: _scrub_snaps_multi: echo 'TEST PASSED' 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1160: _scrub_snaps_multi: return 0 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: err=0 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: return 0 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:13:04.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:13:04.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:13:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:13:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:13:04.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:13:04.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:13:04.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:13:04.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:13:04.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:13:04.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:13:04.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:13:04.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:13:04.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:13:04.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:13:04.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:13:04.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:13:04.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:13:04.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:13:04.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:13:04.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:13:04.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:04.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:04.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.483225 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-snaps 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-snaps 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:13:04.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:13:04.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:13:04.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:13:04.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:13:04.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:13:04.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:13:04.084 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:13:04.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:13:04.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:13:04.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:13:04.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:13:04.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:13:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:13:04.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:13:04.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:13:04.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:13:04.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:13:04.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:13:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:13:04.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:13:04.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:04.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:04.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.483225 2024-02-05T06:13:04.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:13:04.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:13:04.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-snaps 2024-02-05T06:13:04.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:13:04.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:04.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:04.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.483225 2024-02-05T06:13:04.096 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1164: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1165: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1166: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:730: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:731: _scrub_snaps_multi: local poolname=test 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local OBJS=16 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local OSDS=2 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local which=replica 2024-02-05T06:13:04.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: TESTDATA=testdata.483225 2024-02-05T06:13:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2024-02-05T06:13:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-snaps 2024-02-05T06:13:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:13:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:13:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:13:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-snaps/a 2024-02-05T06:13:04.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2024-02-05T06:13:04.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:13:04.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:13:04.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:13:04.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:13:04.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:04.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:04.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:13:04.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:13:04.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:13:04.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:13:04.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:13:04.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:13:04.677 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.483225/ceph-mon.a.asok 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:13:04.678 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.483225/ceph-mon.a.asok config get fsid 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:04.760 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:04.761 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.483225/ceph-mon.a.asok 2024-02-05T06:13:04.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:13:04.761 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.483225/ceph-mon.a.asok config get mon_host 2024-02-05T06:13:04.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:739: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2024-02-05T06:13:04.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-snaps 2024-02-05T06:13:04.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:13:04.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:13:04.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:13:04.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-snaps/x 2024-02-05T06:13:04.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:13:05.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:13:05.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:13:05.685 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:13:05.686 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: seq 0 1 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:13:05.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:13:05.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:13:05.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:13:05.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9c37320-e41a-49b7-9c64-907dd9f8c1e1 2024-02-05T06:13:05.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d9c37320-e41a-49b7-9c64-907dd9f8c1e1' 2024-02-05T06:13:05.693 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 d9c37320-e41a-49b7-9c64-907dd9f8c1e1 2024-02-05T06:13:05.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:13:05.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxfMBlwwhXLBAAXmyx3hoxOWKoLbk2grFwgQ== 2024-02-05T06:13:05.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxfMBlwwhXLBAAXmyx3hoxOWKoLbk2grFwgQ=="}' 2024-02-05T06:13:05.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9c37320-e41a-49b7-9c64-907dd9f8c1e1 -i td/osd-scrub-snaps/0/new.json 2024-02-05T06:13:05.959 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:13:05.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/0/new.json 2024-02-05T06:13:05.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxfMBlwwhXLBAAXmyx3hoxOWKoLbk2grFwgQ== --osd-uuid d9c37320-e41a-49b7-9c64-907dd9f8c1e1 2024-02-05T06:13:06.006 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:06.034+0000 7f734541d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:06.009 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:06.038+0000 7f734541d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:06.010 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:06.039+0000 7f734541d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:06.011 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:06.039+0000 7f734541d4c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:13:06.011 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:06.039+0000 7f734541d4c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:13:06.011 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:06.039+0000 7f734541d4c0 -1 bluestore(td/osd-scrub-snaps/0/block) _read_bdev_label failed to open td/osd-scrub-snaps/0/block: (2) No such file or directory 2024-02-05T06:13:06.185 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:06.213+0000 7f734541d4c0 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2024-02-05T06:13:11.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2024-02-05T06:13:11.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:13:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:13:11.685 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:13:11.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:13:12.054 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:13:12.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:13:12.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:13:12.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:13:12.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:13:12.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:13:12.088 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:12.117+0000 7fbb6f0204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:12.090 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:12.119+0000 7fbb6f0204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:12.091 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:12.120+0000 7fbb6f0204c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:12.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:13:12.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:13:12.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:13:12.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:13:12.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:13:12.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:12.374 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:13:12.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:13:12.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:12.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:12.619 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:12.649+0000 7fbb6f0204c0 -1 Falling back to public interface 2024-02-05T06:13:12.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:13.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:13.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:13.693 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:13:13.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:13:13.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:13.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:13.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:14.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:14.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:14.975 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:13:14.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:13:14.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:14.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:15.001 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:15.030+0000 7fbb6f0204c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:13:15.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:16.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:16.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:16.302 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:13:16.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:13:16.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:16.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:16.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:17.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:17.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:17.589 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:13:17.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:13:17.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:17.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:17.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:18.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:18.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:18.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:13:18.905 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:13:18.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:18.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:19.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:20.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:20.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:20.194 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:13:20.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:13:20.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:20.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:20.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:21.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:21.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:21.483 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:13:21.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:13:21.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:21.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:21.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:22.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:22.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:22.795 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:13:22.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:13:22.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:22.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:23.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:24.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:24.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:24.087 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:13:24.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:13:24.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:24.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:24.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:25.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:25.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:25.375 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:13:25.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:13:25.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:25.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:25.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:26.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:26.778 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:13:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:13:26.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:26.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:27.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:28.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:28.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:28.412 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:13:28.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:13:28.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:28.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:29.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:30.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:30.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:30.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:13:30.300 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:13:30.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:30.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:31.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:32.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:32.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:32.229 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:13:32.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:13:32.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:32.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:32.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:33.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:33.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:33.590 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:13:33.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:13:33.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:33.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:33.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:34.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:34.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:34.896 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:13:34.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:13:34.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:34.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:35.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:36.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:36.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:36.699 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:13:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:13:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:36.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:37.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:38.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:38.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:38.005 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:13:38.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:13:38.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:38.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:13:38.287 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2454085725,v1:127.0.0.1:6803/2454085725] [v2:127.0.0.1:6804/2454085725,v1:127.0.0.1:6805/2454085725] exists,up d9c37320-e41a-49b7-9c64-907dd9f8c1e1 2024-02-05T06:13:38.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:13:38.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:13:38.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:13:38.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:13:38.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:13:38.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fb11b697-7b85-45bb-87f8-e0b211e4fb75 2024-02-05T06:13:38.296 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 fb11b697-7b85-45bb-87f8-e0b211e4fb75 2024-02-05T06:13:38.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fb11b697-7b85-45bb-87f8-e0b211e4fb75' 2024-02-05T06:13:38.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:13:38.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSfMBlQ/B8FBAARLPBm+LYtTws07SMlbL4LA== 2024-02-05T06:13:38.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSfMBlQ/B8FBAARLPBm+LYtTws07SMlbL4LA=="}' 2024-02-05T06:13:38.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fb11b697-7b85-45bb-87f8-e0b211e4fb75 -i td/osd-scrub-snaps/1/new.json 2024-02-05T06:13:38.637 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:13:38.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-snaps/1/new.json 2024-02-05T06:13:38.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSfMBlQ/B8FBAARLPBm+LYtTws07SMlbL4LA== --osd-uuid fb11b697-7b85-45bb-87f8-e0b211e4fb75 2024-02-05T06:13:38.685 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:38.714+0000 7fab968f64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:38.714 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:38.743+0000 7fab968f64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:38.714 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:38.744+0000 7fab968f64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:38.715 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:38.744+0000 7fab968f64c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:13:38.715 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:38.744+0000 7fab968f64c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:13:38.715 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:38.745+0000 7fab968f64c0 -1 bluestore(td/osd-scrub-snaps/1/block) _read_bdev_label failed to open td/osd-scrub-snaps/1/block: (2) No such file or directory 2024-02-05T06:13:38.788 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:38.818+0000 7fab968f64c0 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2024-02-05T06:13:44.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2024-02-05T06:13:44.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:13:44.836 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:13:44.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:13:44.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:13:45.188 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:13:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:13:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:13:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:13:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:13:45.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:13:45.223 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:45.252+0000 7f3e9b14a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:45.225 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:45.254+0000 7f3e9b14a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:45.226 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:45.255+0000 7f3e9b14a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:13:45.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:13:45.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:13:45.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:13:45.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:13:45.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:13:45.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:45.465 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:13:45.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:13:45.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:45.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:45.749 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:45.779+0000 7f3e9b14a4c0 -1 Falling back to public interface 2024-02-05T06:13:45.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:46.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:46.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:46.842 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:13:46.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:13:46.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:46.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:47.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:48.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:48.129 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:13:48.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:48.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:13:48.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:48.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:48.183 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:13:48.212+0000 7f3e9b14a4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:13:48.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:49.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:49.418 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:13:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:13:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:49.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:49.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:50.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:50.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:50.723 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:13:50.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:13:50.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:50.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:51.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:52.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:52.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:52.012 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:13:52.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:13:52.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:52.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:52.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:53.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:53.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:53.299 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:13:53.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:13:53.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:53.300 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:53.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:54.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:54.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:54.583 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:13:54.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:13:54.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:54.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:54.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:55.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:55.871 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:13:55.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:13:55.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:55.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:56.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:57.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:57.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:57.222 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:13:57.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:13:57.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:57.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:57.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:58.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:58.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:58.509 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:13:58.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:13:58.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:58.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:13:58.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:13:59.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:13:59.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:13:59.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:13:59.816 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:13:59.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:13:59.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:00.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:01.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:01.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:01.116 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:14:01.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:14:01.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:01.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:01.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:02.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:02.743 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:14:02.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:02.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:14:02.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:02.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:03.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:04.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:04.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:04.152 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:14:04.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:14:04.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:04.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:04.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:05.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:05.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:05.457 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:14:05.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:14:05.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:05.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:06.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:07.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:07.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:07.182 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:14:07.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:14:07.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:07.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:07.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:08.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:08.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:08.771 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:14:08.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:14:08.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:08.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:09.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:10.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:10.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:10.057 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:14:10.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:14:10.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:10.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:10.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:14:11.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:14:11.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:14:11.923 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:14:11.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:14:11.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:14:11.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:14:12.208 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/458877670,v1:127.0.0.1:6811/458877670] [v2:127.0.0.1:6812/458877670,v1:127.0.0.1:6813/458877670] exists,up fb11b697-7b85-45bb-87f8-e0b211e4fb75 2024-02-05T06:14:12.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:14:12.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:14:12.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:14:12.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:746: _scrub_snaps_multi: ceph osd set noscrub 2024-02-05T06:14:12.583 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:14:12.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:747: _scrub_snaps_multi: ceph osd set nodeep-scrub 2024-02-05T06:14:12.968 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:14:12.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:750: _scrub_snaps_multi: create_pool test 1 1 2024-02-05T06:14:12.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:14:13.462 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T06:14:13.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:14:14.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:751: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:14:14.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:14:14.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:14:14.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:14:14.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:14:14.476 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:14:14.476 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:14:14.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:14:14.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:14:14.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:14:14.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:14:14.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:14:14.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:14:14.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:14:14.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:14:14.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:14:14.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:14:14.838 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T06:14:14.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:14:14.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:14:14.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:14:14.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-02-05T06:14:14.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-02-05T06:14:14.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-02-05T06:14:14.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:14:14.950 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:14:15.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-02-05T06:14:15.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-02-05T06:14:15.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672972' 2024-02-05T06:14:15.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:14:15.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-02-05T06:14:15.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:14:15.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:14:15.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-02-05T06:14:15.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:14:15.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-02-05T06:14:15.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-02-05T06:14:15.065 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836490 2024-02-05T06:14:15.065 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:14:15.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-02-05T06:14:15.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:14:16.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:14:16.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:14:16.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-02-05T06:14:16.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:14:16.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-02-05T06:14:16.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:14:16.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:14:16.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-02-05T06:14:16.625 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:14:16.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-02-05T06:14:16.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-02-05T06:14:16.626 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672972 2024-02-05T06:14:16.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:14:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-02-05T06:14:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:14:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:14:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:14:17.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:14:17.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:14:17.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:14:17.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:14:17.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:14:17.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:14:17.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:14:17.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:14:17.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: ceph osd dump 2024-02-05T06:14:17.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2024-02-05T06:14:17.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: awk '{ print $2 }' 2024-02-05T06:14:18.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: poolid=1 2024-02-05T06:14:18.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.483225 bs=1032 count=1 2024-02-05T06:14:18.259 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:14:18.259 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:14:18.260 INFO:tasks.workunit.client.0.smithi188.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.481e-05 s, 23.0 MB/s 2024-02-05T06:14:18.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: seq 1 16 2024-02-05T06:14:18.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:18.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj1 testdata.483225 2024-02-05T06:14:18.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:18.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj2 testdata.483225 2024-02-05T06:14:18.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:18.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj3 testdata.483225 2024-02-05T06:14:18.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:18.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj4 testdata.483225 2024-02-05T06:14:18.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:18.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj5 testdata.483225 2024-02-05T06:14:18.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:18.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj6 testdata.483225 2024-02-05T06:14:19.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:19.049 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj7 testdata.483225 2024-02-05T06:14:19.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:19.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj8 testdata.483225 2024-02-05T06:14:19.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:19.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj9 testdata.483225 2024-02-05T06:14:19.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:19.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj10 testdata.483225 2024-02-05T06:14:19.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:19.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj11 testdata.483225 2024-02-05T06:14:19.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:19.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj12 testdata.483225 2024-02-05T06:14:19.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:19.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj13 testdata.483225 2024-02-05T06:14:20.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:20.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj14 testdata.483225 2024-02-05T06:14:20.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:20.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj15 testdata.483225 2024-02-05T06:14:20.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:755: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2024-02-05T06:14:20.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: rados -p test put obj16 testdata.483225 2024-02-05T06:14:20.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: get_primary test obj1 2024-02-05T06:14:20.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:14:20.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:14:20.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:14:20.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:14:20.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:760: _scrub_snaps_multi: local primary=1 2024-02-05T06:14:20.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: get_not_primary test obj1 2024-02-05T06:14:20.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-02-05T06:14:20.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-02-05T06:14:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-02-05T06:14:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:14:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:14:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:14:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:14:21.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-02-05T06:14:21.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-02-05T06:14:21.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:761: _scrub_snaps_multi: local replica=0 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.483225 '$replica' 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.483225 0 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.483225 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2024-02-05T06:14:21.341 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2024-02-05T06:14:21.607 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap1 2024-02-05T06:14:21.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=1 2024-02-05T06:14:21.610 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:14:21.610 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:14:21.610 INFO:tasks.workunit.client.0.smithi188.stderr:256 bytes copied, 5.9318e-05 s, 4.3 MB/s 2024-02-05T06:14:21.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.483225 2024-02-05T06:14:21.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:14:21.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.483225 2024-02-05T06:14:22.188 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2024-02-05T06:14:22.188 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:22.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.483225 2024-02-05T06:14:22.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:22.322 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.483225 2024-02-05T06:14:22.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:22.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.483225 2024-02-05T06:14:22.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:22.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.483225 2024-02-05T06:14:22.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:22.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.483225 2024-02-05T06:14:22.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:22.858 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.483225 2024-02-05T06:14:23.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:23.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.483225 2024-02-05T06:14:23.134 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:23.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.483225 2024-02-05T06:14:23.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2024-02-05T06:14:23.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.483225 2024-02-05T06:14:23.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2024-02-05T06:14:23.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2024-02-05T06:14:23.544 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap2 2024-02-05T06:14:23.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=2 2024-02-05T06:14:23.546 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records in 2024-02-05T06:14:23.546 INFO:tasks.workunit.client.0.smithi188.stderr:2+0 records out 2024-02-05T06:14:23.547 INFO:tasks.workunit.client.0.smithi188.stderr:512 bytes copied, 6.114e-05 s, 8.4 MB/s 2024-02-05T06:14:23.547 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:14:23.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2024-02-05T06:14:23.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2024-02-05T06:14:24.028 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap3 2024-02-05T06:14:24.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=3 2024-02-05T06:14:24.031 INFO:tasks.workunit.client.0.smithi188.stderr:3+0 records in 2024-02-05T06:14:24.031 INFO:tasks.workunit.client.0.smithi188.stderr:3+0 records out 2024-02-05T06:14:24.031 INFO:tasks.workunit.client.0.smithi188.stderr:768 bytes copied, 5.9381e-05 s, 12.9 MB/s 2024-02-05T06:14:24.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.483225 2024-02-05T06:14:24.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2024-02-05T06:14:24.249 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2024-02-05T06:14:24.424 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap4 2024-02-05T06:14:24.425 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=4 2024-02-05T06:14:24.426 INFO:tasks.workunit.client.0.smithi188.stderr:4+0 records in 2024-02-05T06:14:24.426 INFO:tasks.workunit.client.0.smithi188.stderr:4+0 records out 2024-02-05T06:14:24.427 INFO:tasks.workunit.client.0.smithi188.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 6.76e-05 s, 15.1 MB/s 2024-02-05T06:14:24.427 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:14:24.685 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.483225 2024-02-05T06:14:24.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2024-02-05T06:14:24.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2024-02-05T06:14:24.969 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap5 2024-02-05T06:14:24.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2024-02-05T06:14:24.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2024-02-05T06:14:25.220 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap6 2024-02-05T06:14:25.222 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=6 2024-02-05T06:14:25.223 INFO:tasks.workunit.client.0.smithi188.stderr:6+0 records in 2024-02-05T06:14:25.223 INFO:tasks.workunit.client.0.smithi188.stderr:6+0 records out 2024-02-05T06:14:25.223 INFO:tasks.workunit.client.0.smithi188.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 7.2288e-05 s, 21.2 MB/s 2024-02-05T06:14:25.224 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.483225 2024-02-05T06:14:25.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2024-02-05T06:14:25.506 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2024-02-05T06:14:25.729 INFO:tasks.workunit.client.0.smithi188.stdout:created pool test snap snap7 2024-02-05T06:14:25.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2024-02-05T06:14:25.966 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2024-02-05T06:14:26.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2024-02-05T06:14:26.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2024-02-05T06:14:26.210 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:14:26.210 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:14:26.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:14:26.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:14:26.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:14:26.516 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:14:26.517 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2024-02-05T06:14:27.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:27.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2024-02-05T06:14:28.658 INFO:tasks.workunit.client.0.smithi188.stdout:WARNING: only removing head with clones present 2024-02-05T06:14:28.659 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:ff7b1f36:::obj1:head# 2024-02-05T06:14:29.174 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:14:29.174 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2024-02-05T06:14:30.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:30.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:14:30.795 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:c52c9666:::obj5:2# 2024-02-05T06:14:31.310 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:14:31.310 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2024-02-05T06:14:32.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:32.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:32.414 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:14:33.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:14:33.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:33.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:14:33.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:33.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:14:33.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:14:33.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:33.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:14:33.215 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:14:33.215 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:14:33.867 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:c52c9666:::obj5:1# 2024-02-05T06:14:34.382 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:14:35.428 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:14:35.429 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:35.430 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:35.431 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:35.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:14:35.432 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:14:35.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:14:35.432 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2024-02-05T06:14:35.433 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2024-02-05T06:14:36.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:36.319 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=18 2024-02-05T06:14:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:18+0 records in 2024-02-05T06:14:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:18+0 records out 2024-02-05T06:14:36.320 INFO:tasks.workunit.client.0.smithi188.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000142355 s, 32.4 MB/s 2024-02-05T06:14:36.320 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:14:37.424 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2024-02-05T06:14:39.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:39.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=15 2024-02-05T06:14:39.020 INFO:tasks.workunit.client.0.smithi188.stderr:15+0 records in 2024-02-05T06:14:39.020 INFO:tasks.workunit.client.0.smithi188.stderr:15+0 records out 2024-02-05T06:14:39.020 INFO:tasks.workunit.client.0.smithi188.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000103025 s, 37.3 MB/s 2024-02-05T06:14:39.020 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:14:40.110 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2024-02-05T06:14:40.110 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2024-02-05T06:14:41.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:41.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:14:41.640 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:0ee9ae15:::obj4:7# 2024-02-05T06:14:42.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:14:43.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:14:43.210 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:43.211 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:43.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2024-02-05T06:14:43.213 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2024-02-05T06:14:43.214 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2024-02-05T06:14:44.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:44.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2024-02-05T06:14:44.621 INFO:tasks.workunit.client.0.smithi188.stdout:remove #1:461f8b5e:::obj16:7# 2024-02-05T06:14:45.136 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2024-02-05T06:14:46.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:46.187 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2024-02-05T06:14:46.188 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:46.189 INFO:tasks.workunit.client.0.smithi188.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2024-02-05T06:14:46.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2024-02-05T06:14:46.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2024-02-05T06:14:46.189 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2024-02-05T06:14:46.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:46.977 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2024-02-05T06:14:48.050 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:48.050 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2024-02-05T06:14:48.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:48.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.483225 bs=256 count=7 2024-02-05T06:14:48.053 INFO:tasks.workunit.client.0.smithi188.stderr:7+0 records in 2024-02-05T06:14:48.053 INFO:tasks.workunit.client.0.smithi188.stderr:7+0 records out 2024-02-05T06:14:48.053 INFO:tasks.workunit.client.0.smithi188.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 9.4838e-05 s, 18.9 MB/s 2024-02-05T06:14:48.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.483225 2024-02-05T06:14:49.477 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2024-02-05T06:14:50.207 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:14:50.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:50.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2024-02-05T06:14:51.536 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2024-02-05T06:14:52.244 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:14:53.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:53.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2024-02-05T06:14:54.083 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2024-02-05T06:14:54.732 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:14:54.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:54.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2024-02-05T06:14:56.064 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2024-02-05T06:14:56.794 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:14:57.559 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:57.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2024-02-05T06:14:58.595 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2024-02-05T06:14:59.240 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:14:59.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:14:59.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2024-02-05T06:15:00.598 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2024-02-05T06:15:01.264 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:15:01.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:15:01.528 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2024-02-05T06:15:02.632 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2024-02-05T06:15:03.289 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:15:03.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:15:03.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2024-02-05T06:15:04.612 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2024-02-05T06:15:05.275 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:15:05.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:15:05.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2024-02-05T06:15:06.583 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2024-02-05T06:15:07.254 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:15:08.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:15:08.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2024-02-05T06:15:09.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2024-02-05T06:15:09.088 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2024-02-05T06:15:09.792 INFO:tasks.workunit.client.0.smithi188.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2024-02-05T06:15:10.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2024-02-05T06:15:10.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2024-02-05T06:15:11.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2024-02-05T06:15:11.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2024-02-05T06:15:11.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: rm -f testdata.483225 2024-02-05T06:15:11.088 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: expr 2 - 1 2024-02-05T06:15:11.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: seq 0 1 2024-02-05T06:15:11.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/0 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:15:11.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:15:11.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:15:11.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:15:11.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:15:11.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:15:11.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/0 2024-02-05T06:15:11.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-02-05T06:15:11.096 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:15:11.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:15:11.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/0/whoami 2024-02-05T06:15:11.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-02-05T06:15:11.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:15:11.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:15:11.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:15:11.131 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:11.161+0000 7f26c14c04c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:11.133 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:11.164+0000 7f26c14c04c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:11.134 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:11.165+0000 7f26c14c04c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:15:11.379 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:15:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:11.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:11.649 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:11.680+0000 7f26c14c04c0 -1 Falling back to public interface 2024-02-05T06:15:11.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:12.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:12.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:12.673 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:15:12.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:15:12.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:12.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:12.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:13.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:13.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:13.951 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:15:13.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:15:13.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:13.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:14.023 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:14.053+0000 7f26c14c04c0 -1 osd.0 23 log_to_monitors true 2024-02-05T06:15:14.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:15.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:15.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:15.255 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:15:15.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:15:15.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:15.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:15.527 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 24 last_clean_interval [5,23) [v2:127.0.0.1:6802/1512949593,v1:127.0.0.1:6803/1512949593] [v2:127.0.0.1:6804/1512949593,v1:127.0.0.1:6805/1512949593] exists,up d9c37320-e41a-49b7-9c64-907dd9f8c1e1 2024-02-05T06:15:15.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:15:15.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:15:15.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:15:15.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:15:15.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2024-02-05T06:15:15.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-snaps 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-snaps/1 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1' 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-02-05T06:15:15.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:15:15.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:15:15.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:15:15.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-02-05T06:15:15.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-02-05T06:15:15.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-snaps/1 2024-02-05T06:15:15.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-02-05T06:15:15.608 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:15:15.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.483225/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-02-05T06:15:15.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-snaps/1/whoami 2024-02-05T06:15:15.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-02-05T06:15:15.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-02-05T06:15:15.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-02-05T06:15:15.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-02-05T06:15:15.610 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:15.594+0000 7fccafb1e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:15.610 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:15.596+0000 7fccafb1e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:15.610 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:15.597+0000 7fccafb1e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:15.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-02-05T06:15:15.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:15:15.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:15:15.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:15:15.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:15:15.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:15.881 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:15:15.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:15:15.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:15.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:16.084 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:16.114+0000 7fccafb1e4c0 -1 Falling back to public interface 2024-02-05T06:15:16.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:17.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:17.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:17.160 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:15:17.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:15:17.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:17.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:17.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:18.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:18.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:18.448 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:15:18.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:15:18.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:18.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:18.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:18.770 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:18.800+0000 7fccafb1e4c0 -1 osd.1 23 log_to_monitors true 2024-02-05T06:15:19.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:19.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:19.726 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:15:19.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:15:19.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:19.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:20.010 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/587627540,v1:127.0.0.1:6811/587627540] [v2:127.0.0.1:6812/587627540,v1:127.0.0.1:6813/587627540] exists,up fb11b697-7b85-45bb-87f8-e0b211e4fb75 2024-02-05T06:15:20.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:15:20.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:15:20.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:15:20.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:772: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2024-02-05T06:15:20.115 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:15:20.115 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:15:20.115 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.129 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:15:20.129 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' (not observed, change may require restart) osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-02-05T06:15:20.130 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:773: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2024-02-05T06:15:20.278 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:15:20.278 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:15:20.278 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.292 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:15:20.292 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_shallow_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:15:20.293 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2024-02-05T06:15:20.439 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:15:20.439 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:15:20.439 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.453 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:15:20.453 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_chunk_min = '' (not observed, change may require restart) " 2024-02-05T06:15:20.453 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max 1 2024-02-05T06:15:20.598 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:15:20.598 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:15:20.599 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.612 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:15:20.612 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_pg_stat_report_interval_max = '' (not observed, change may require restart) " 2024-02-05T06:15:20.612 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:20.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: wait_for_clean 2024-02-05T06:15:20.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:15:20.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:15:20.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:15:20.622 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:15:20.623 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:15:20.623 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:15:20.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:15:20.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:15:20.623 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:15:20.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:15:20.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:15:20.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:15:20.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:15:20.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:15:20.706 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:15:20.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:15:20.980 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T06:15:20.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:15:20.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:15:20.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:15:21.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-02-05T06:15:21.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-02-05T06:15:21.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699' 2024-02-05T06:15:21.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:15:21.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:15:21.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-02-05T06:15:21.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-02-05T06:15:21.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149699 1-124554051586' 2024-02-05T06:15:21.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:15:21.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149699 2024-02-05T06:15:21.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:15:21.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:15:21.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149699 2024-02-05T06:15:21.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:15:21.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-02-05T06:15:21.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2024-02-05T06:15:21.205 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 111669149699 2024-02-05T06:15:21.206 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:15:21.478 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149699 2024-02-05T06:15:21.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:15:22.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:15:22.480 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:15:22.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-02-05T06:15:22.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:15:22.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-02-05T06:15:22.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:15:22.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:15:22.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-02-05T06:15:22.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:15:22.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-02-05T06:15:22.764 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-02-05T06:15:22.765 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 124554051586 2024-02-05T06:15:22.765 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:15:23.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-02-05T06:15:23.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:15:23.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:23.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:23.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:15:23.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:15:23.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:15:23.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:15:23.422 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:15:23.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:15:23.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:15:23.423 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:15:23.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:15:23.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:15:23.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:15:23.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: local pgid=1.0 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: pg_scrub 1.0 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=1.0 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 1.0 2024-02-05T06:15:24.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:15:24.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:24.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:24.048 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:15:24.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:24.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 1.0 2024-02-05T06:15:24.578 INFO:tasks.workunit.client.0.smithi188.stderr:instructing pg 1.0 on osd.1 to scrub 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 1.0 2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:24.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:15:24.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:14:13.438573+0000 '>' 2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:24.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:15:25.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:15:25.867 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:25.868 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:15:25.868 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:15:25.868 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:25.868 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:25.868 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:15:26.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:14:13.438573+0000 '>' 2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:26.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:15:27.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:15:27.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:27.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:15:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:15:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:15:27.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:14:13.438573+0000 '>' 2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:27.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:15:28.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:15:28.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:28.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:15:28.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:15:28.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:28.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:28.429 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:15:28.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:14:13.438573+0000 '>' 2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:28.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:15:29.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:15:29.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-02-05T06:15:29.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:15:29.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:15:29.709 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:15:29.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:15:29.710 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:15:29.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:15:25.507710+0000 '>' 2024-02-05T06:14:13.438573+0000 2024-02-05T06:15:29.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:15:29.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:29.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: wc -l 2024-02-05T06:15:29.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:783: _scrub_snaps_multi: test 16 -gt 3 2024-02-05T06:15:29.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:15:29.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: wc -l 2024-02-05T06:15:29.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:784: _scrub_snaps_multi: test 16 -gt 3 2024-02-05T06:15:29.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: rados list-inconsistent-pg test 2024-02-05T06:15:30.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2024-02-05T06:15:30.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:788: _scrub_snaps_multi: test 1 = 1 2024-02-05T06:15:30.033 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2024-02-05T06:15:30.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:790: _scrub_snaps_multi: test 1.0 = 1.0 2024-02-05T06:15:30.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:792: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2024-02-05T06:15:30.068 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:15:30.068 INFO:tasks.workunit.client.0.smithi188.stdout: "epoch": 29, 2024-02-05T06:15:30.068 INFO:tasks.workunit.client.0.smithi188.stdout: "inconsistents": [ 2024-02-05T06:15:30.068 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.068 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.068 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:15:30.068 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 5 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.069 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 1, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'19", 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'5", 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4243.0:1", 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 5, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:18.860667+0000", 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:18.861188+0000", 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.070 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x2b1d9977", 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.071 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.072 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 2, 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 20 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.073 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 2, 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "18'41", 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "17'20", 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4291.0:1", 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 20, 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:21.926910+0000", 2024-02-05T06:15:30.074 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:21.927458+0000", 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.075 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.076 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 4, 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 42 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch" 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.077 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 4, 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "20'45", 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "18'42", 2024-02-05T06:15:30.078 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4327.0:1", 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 42, 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 512, 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:23.610245+0000", 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:23.632382+0000", 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x64017b98", 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.079 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.080 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 4608, 2024-02-05T06:15:30.081 INFO:tasks.workunit.client.0.smithi188.stdout: "object_info": { 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj5", 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 4, 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1718170787, 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.083 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "20'45", 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "18'42", 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4327.0:1", 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 42, 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 512, 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:23.610245+0000", 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:23.632382+0000", 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.084 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x64017b98", 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.085 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 512 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj4", 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 7, 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 4 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.086 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj4", 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": 7, 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2826278768, 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "23'51", 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'4", 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4240.0:1", 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 4, 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.087 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:18.727511+0000", 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:18.728071+0000", 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x2b1d9977", 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.088 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.089 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj5", 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 7, 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 0 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.090 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "missing", 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "info_missing" 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "info_missing" 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1792 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.091 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj1", 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 18 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.092 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj1", 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1828249343, 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'18", 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'1", 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4288.0:1", 2024-02-05T06:15:30.093 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 18, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:21.674062+0000", 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:21.756128+0000", 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.094 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "missing" 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.095 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj10", 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 32 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.096 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj10", 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 718195851, 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'32", 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'10", 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4309.0:1", 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 32, 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:22.770684+0000", 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:22.771243+0000", 2024-02-05T06:15:30.097 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.098 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "????", 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.099 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.100 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj11", 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.101 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 34 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj11", 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 693400951, 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.102 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'34", 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'11", 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4312.0:1", 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 34, 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:22.920861+0000", 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:22.921436+0000", 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.103 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.104 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.105 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.106 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj13", 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 38 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj13", 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2087409765, 2024-02-05T06:15:30.107 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'38", 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'13", 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4318.0:1", 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 38, 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:23.197437+0000", 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:23.197983+0000", 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.108 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.109 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]" 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.110 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.111 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj14", 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 40 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.112 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj14", 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2484217095, 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'40", 2024-02-05T06:15:30.113 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'14", 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4321.0:1", 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 40, 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:23.323399+0000", 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:23.323942+0000", 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.114 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.115 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1033, 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.116 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.117 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj15", 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 15 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_corrupted" 2024-02-05T06:15:30.118 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj15", 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 612772309, 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "16'15", 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "0'0", 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4273.0:1", 2024-02-05T06:15:30.119 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 15, 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:20.218542+0000", 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:20.219082+0000", 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x2b1d9977", 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.120 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_corrupted" 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.121 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": "Z2FyYmFnZQo=" 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.122 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj2", 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 56 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_missing" 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj2", 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.123 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1058988552, 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "23'56", 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "20'48", 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4363.0:1", 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 56, 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 0, 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:26.162382+0000", 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:26.162926+0000", 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "whiteout", 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty" 2024-02-05T06:15:30.124 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xffffffff", 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.125 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_missing" 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 0 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 0, 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 7, 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.126 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 4, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: 4, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: 3, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: 2, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 7, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1024, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: 7, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: 6, 2024-02-05T06:15:30.127 INFO:tasks.workunit.client.0.smithi188.stdout: 5 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj3", 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 44 2024-02-05T06:15:30.128 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch" 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [ 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj3", 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1643547569, 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.129 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "19'44", 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "17'22", 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4333.0:1", 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 44, 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 768, 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:24.095007+0000", 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:24.101348+0000", 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.130 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xb462972e", 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.131 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "size_mismatch_info", 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "obj_size_info_mismatch" 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 3840, 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "object_info": { 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj3", 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1643547569, 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.132 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "19'44", 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "17'22", 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4333.0:1", 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 44, 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 768, 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:24.095007+0000", 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:24.101348+0000", 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0xb462972e", 2024-02-05T06:15:30.133 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 768 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.134 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj6", 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 24 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj6", 2024-02-05T06:15:30.135 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2202164420, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'24", 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'6", 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4297.0:1", 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 24, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:22.253232+0000", 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:22.253737+0000", 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.136 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.137 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.138 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.139 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj7", 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 26 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj7", 2024-02-05T06:15:30.140 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 1552453721, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'26", 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'7", 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4300.0:1", 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 26, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:22.385628+0000", 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:22.386196+0000", 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.141 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.142 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [] 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.143 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.144 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj8", 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 28 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj8", 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.145 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 2381834917, 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'28", 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'8", 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4303.0:1", 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 28, 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:22.519694+0000", 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:22.520254+0000", 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.146 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.147 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 0, 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.148 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.149 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: "object": { 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: "name": "obj9", 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: "nspace": "", 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: "locator": "", 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": "head", 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: "version": 30 2024-02-05T06:15:30.150 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [ 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset_inconsistency" 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "union_shard_errors": [], 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "selected_object_info": { 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": { 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "oid": "obj9", 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "key": "", 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "snapid": -2, 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "hash": 3833113727, 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "max": 0, 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "pool": 1, 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "namespace": "" 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "version": "17'30", 2024-02-05T06:15:30.151 INFO:tasks.workunit.client.0.smithi188.stdout: "prior_version": "16'9", 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "last_reqid": "client.4306.0:1", 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "user_version": 30, 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "mtime": "2024-02-05T06:14:22.644633+0000", 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "local_mtime": "2024-02-05T06:14:22.645214+0000", 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "lost": 0, 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "flags": [ 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "dirty", 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest" 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_seq": 0, 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "truncate_size": 0, 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "data_digest": "0x14f55900", 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "omap_digest": "0xffffffff", 2024-02-05T06:15:30.152 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_object_size": 0, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "expected_write_size": 0, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "alloc_hint_flags": 0, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "manifest": { 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "type": 0 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "watchers": {} 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "shards": [ 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 0, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": false, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.153 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: "size": "????", 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: "osd": 1, 2024-02-05T06:15:30.154 INFO:tasks.workunit.client.0.smithi188.stdout: "primary": true, 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "errors": [], 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 256, 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "snapset": { 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "seq": 1, 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "clones": [ 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "snap": 1, 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "size": 1032, 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "overlap": "[]", 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: "snaps": [ 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: 1 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.155 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.156 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.156 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:15:30.156 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:15:30.156 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:15:30.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:794: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2024-02-05T06:15:30.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: '[' replica = replica ']' 2024-02-05T06:15:30.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:799: _scrub_snaps_multi: scruberrors=20 2024-02-05T06:15:30.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: jq .inconsistents 2024-02-05T06:15:30.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:15:30.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1063: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1064: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1065: _scrub_snaps_multi: test no = yes 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1070: _scrub_snaps_multi: test '' = yes 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: find td/osd-scrub-snaps 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1075: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:td/osd-scrub-snaps/osd.0.pid' 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1076: _scrub_snaps_multi: pids= 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='547646 ' 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1077: _scrub_snaps_multi: for pidfile in ${pidfiles} 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2024-02-05T06:15:30.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids+='547082 ' 2024-02-05T06:15:30.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: ERRORS=0 2024-02-05T06:15:30.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1086: _scrub_snaps_multi: '[' replica = primary ']' 2024-02-05T06:15:30.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1108: _scrub_snaps_multi: ceph pg dump pgs 2024-02-05T06:15:30.413 INFO:tasks.workunit.client.0.smithi188.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-02-05T06:15:30.413 INFO:tasks.workunit.client.0.smithi188.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2024-02-05T06:15:25.507818+0000 23'56 30:95 [1,0] 1 [1,0] 1 23'56 2024-02-05T06:15:25.507710+0000 0'0 2024-02-05T06:14:13.438573+0000 0 1 periodic scrub scheduled @ 2024-02-06T15:12:19.947959+0000 36 0 2024-02-05T06:15:30.413 INFO:tasks.workunit.client.0.smithi188.stdout: 2024-02-05T06:15:30.413 INFO:tasks.workunit.client.0.smithi188.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-02-05T06:15:30.413 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:15:30.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:15:30.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 547646 2024-02-05T06:15:30.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1110: _scrub_snaps_multi: for pid in $pids 2024-02-05T06:15:30.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1112: _scrub_snaps_multi: kill -0 547082 2024-02-05T06:15:30.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1119: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2024-02-05T06:15:30.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:15:30.425 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:15:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:15:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:15:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:15:35.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:15:35.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1121: _scrub_snaps_multi: declare -a err_strings 2024-02-05T06:15:35.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2024-02-05T06:15:35.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1123: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.750 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2024-02-05T06:15:35.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1133: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2024-02-05T06:15:35.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1141: _scrub_snaps_multi: declare -a rep_err_strings 2024-02-05T06:15:35.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: eval echo '$replica' 2024-02-05T06:15:35.759 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: echo 0 2024-02-05T06:15:35.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1142: _scrub_snaps_multi: osd=0 2024-02-05T06:15:35.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1143: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2024-02-05T06:15:35.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2024-02-05T06:15:35.759 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2024-02-05T06:15:35.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1153: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2024-02-05T06:15:35.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1159: _scrub_snaps_multi: echo 'TEST PASSED' 2024-02-05T06:15:35.761 INFO:tasks.workunit.client.0.smithi188.stdout:TEST PASSED 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1160: _scrub_snaps_multi: return 0 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: err=0 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=990fa250-2137-4da0-9ced-ff79af2e8903 --auth-supported=none --mon-host=127.0.0.1:7121 ' 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: return 0 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:15:35.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:15:35.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:15:35.768 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:15:35.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:15:35.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:15:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:15:35.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:15:35.771 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:15:35.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:35.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:15:35.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:15:35.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:35.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:15:35.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:15:35.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:15:35.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:15:35.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:35.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:15:35.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.483225 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-snaps 0 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-snaps 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-snaps KILL 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:15:35.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:15:35.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:15:35.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:15:35.793 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:15:35.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:15:35.794 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:15:35.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:15:35.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:15:35.796 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:15:35.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:35.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:15:35.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:15:35.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:35.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:15:35.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-02-05T06:15:35.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-snaps 2024-02-05T06:15:35.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:15:35.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:35.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.483225 2024-02-05T06:15:35.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.483225 2024-02-05T06:15:35.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:15:35.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:15:35.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-02-05T06:15:35.803 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-02-05T06:15:35.803 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T06:15:35.831 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2024-02-05T06:15:35.831 DEBUG:teuthology.orchestra.run.smithi188:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2024-02-05T06:15:35.893 INFO:tasks.workunit.client.0.smithi188.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-02-05T06:15:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-02-05T06:15:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-test 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2024-02-05T06:15:35.902 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2024-02-05T06:15:35.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none ' 2024-02-05T06:15:35.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2024-02-05T06:15:35.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-02-05T06:15:35.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2024-02-05T06:15:35.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-02-05T06:15:35.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_deep_scrub_abort 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_dump_scrub_schedule 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_interval_changes 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_just_deep_scrubs 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_pg_dump_objects_scrubbed 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_scrub_abort 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_scrub_extended_sleep 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_scrub_permit_time 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:TEST_scrub_test' 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2024-02-05T06:15:35.904 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2024-02-05T06:15:35.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T06:15:35.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T06:15:35.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:15:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:15:35.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:15:35.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:35.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:15:35.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:15:35.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:15:35.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:15:35.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:15:35.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:15:35.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:15:35.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:35.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:35.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:15:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:15:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:15:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T06:15:35.912 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:15:35.913 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:35.913 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:35.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.548953 2024-02-05T06:15:35.914 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:15:35.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-02-05T06:15:35.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-02-05T06:15:35.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:15:35.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T06:15:35.914 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2024-02-05T06:15:35.915 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:415: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:298: _scrub_abort: local dir=td/osd-scrub-test 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: local poolname=test 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: local OSDS=3 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:301: _scrub_abort: local objects=1000 2024-02-05T06:15:35.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: local type=deep-scrub 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:304: _scrub_abort: TESTDATA=testdata.548953 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:305: _scrub_abort: test deep-scrub = scrub 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: stopscrub=nodeep-scrub 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: check=nodeep_scrub 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:314: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-02-05T06:15:35.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-02-05T06:15:36.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:15:36.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:36.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:36.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:36.296 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:36.297 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:36.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:36.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-02-05T06:15:36.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:15:36.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:15:36.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:15:36.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:15:36.499 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:36.500 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-mon.a.asok 2024-02-05T06:15:36.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:15:36.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.548953/ceph-mon.a.asok config get fsid 2024-02-05T06:15:36.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-mon.a.asok 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:15:36.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.548953/ceph-mon.a.asok config get mon_host 2024-02-05T06:15:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:315: _scrub_abort: run_mgr td/osd-scrub-test x 2024-02-05T06:15:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-02-05T06:15:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:15:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:15:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:15:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-02-05T06:15:36.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:15:37.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:15:37.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:37.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:37.474 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:37.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:37.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:15:37.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:15:37.500 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: expr 3 - 1 2024-02-05T06:15:37.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: seq 0 2 2024-02-05T06:15:37.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:15:37.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:15:37.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:15:37.508 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 dc43ab43-a024-4f0b-964b-0ac58f50b42b 2024-02-05T06:15:37.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc43ab43-a024-4f0b-964b-0ac58f50b42b 2024-02-05T06:15:37.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dc43ab43-a024-4f0b-964b-0ac58f50b42b' 2024-02-05T06:15:37.509 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:15:37.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJfcBl2KjqIBAAc8wMRHb70O8rvX4vt6/CKw== 2024-02-05T06:15:37.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJfcBl2KjqIBAAc8wMRHb70O8rvX4vt6/CKw=="}' 2024-02-05T06:15:37.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc43ab43-a024-4f0b-964b-0ac58f50b42b -i td/osd-scrub-test/0/new.json 2024-02-05T06:15:37.722 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:15:37.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-02-05T06:15:37.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAJfcBl2KjqIBAAc8wMRHb70O8rvX4vt6/CKw== --osd-uuid dc43ab43-a024-4f0b-964b-0ac58f50b42b 2024-02-05T06:15:37.764 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:37.794+0000 7fd7780134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:37.773 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:37.803+0000 7fd7780134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:37.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:37.804+0000 7fd7780134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:37.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:37.805+0000 7fd7780134c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:15:37.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:37.805+0000 7fd7780134c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:15:37.774 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:37.805+0000 7fd7780134c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:15:37.865 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:37.896+0000 7fd7780134c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T06:15:43.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-02-05T06:15:43.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:15:43.449 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:15:43.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:15:43.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:15:43.813 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:15:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:15:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:15:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:15:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:15:43.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:15:43.841 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:43.871+0000 7f81774144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:43.842 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:43.873+0000 7f81774144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:43.844 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:43.874+0000 7f81774144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:44.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:15:44.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:15:44.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:15:44.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:15:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:15:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:15:44.078 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:15:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:44.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:44.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:44.364 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:44.395+0000 7f81774144c0 -1 Falling back to public interface 2024-02-05T06:15:45.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:45.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:45.341 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:15:45.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:15:45.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:45.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:45.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:46.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:46.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:46.620 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:15:46.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:15:46.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:46.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:46.733 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:46.764+0000 7f81774144c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:15:46.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:47.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:47.909 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:15:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:15:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:47.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:15:48.183 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2329718660,v1:127.0.0.1:6803/2329718660] [v2:127.0.0.1:6804/2329718660,v1:127.0.0.1:6805/2329718660] exists,up dc43ab43-a024-4f0b-964b-0ac58f50b42b 2024-02-05T06:15:48.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:15:48.188 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:48.189 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:15:48.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8eda2309-d1dc-4246-b87d-040183673a45 2024-02-05T06:15:48.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8eda2309-d1dc-4246-b87d-040183673a45' 2024-02-05T06:15:48.191 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 8eda2309-d1dc-4246-b87d-040183673a45 2024-02-05T06:15:48.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:15:48.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUfcBlPIjlDRAA+tkdJLWqCJ2hAG1wJCATTA== 2024-02-05T06:15:48.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUfcBlPIjlDRAA+tkdJLWqCJ2hAG1wJCATTA=="}' 2024-02-05T06:15:48.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8eda2309-d1dc-4246-b87d-040183673a45 -i td/osd-scrub-test/1/new.json 2024-02-05T06:15:48.512 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:15:48.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-02-05T06:15:48.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAUfcBlPIjlDRAA+tkdJLWqCJ2hAG1wJCATTA== --osd-uuid 8eda2309-d1dc-4246-b87d-040183673a45 2024-02-05T06:15:48.552 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:48.583+0000 7f7037a134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:48.596 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:48.627+0000 7f7037a134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:48.597 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:48.628+0000 7f7037a134c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:48.597 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:48.628+0000 7f7037a134c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:15:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:48.628+0000 7f7037a134c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:15:48.598 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:48.628+0000 7f7037a134c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:15:48.671 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:48.702+0000 7f7037a134c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T06:15:54.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-02-05T06:15:54.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:15:54.463 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:15:54.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:15:54.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:15:54.805 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:15:54.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:15:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:15:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:15:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:15:54.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:15:54.832 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:54.862+0000 7f73372144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:54.864+0000 7f73372144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:54.834 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:54.865+0000 7f73372144c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:15:55.069 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:15:55.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:55.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:55.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:55.360 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:55.391+0000 7f73372144c0 -1 Falling back to public interface 2024-02-05T06:15:56.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:56.349 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:15:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:15:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:56.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:56.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:57.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:57.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:57.623 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:15:57.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:15:57.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:57.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:57.847 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:57.878+0000 7f73372144c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:15:57.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:15:58.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:15:58.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:15:58.903 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:15:58.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:15:58.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:15:58.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:15:59.169 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4194562422,v1:127.0.0.1:6811/4194562422] [v2:127.0.0.1:6812/4194562422,v1:127.0.0.1:6813/4194562422] exists,up 8eda2309-d1dc-4246-b87d-040183673a45 2024-02-05T06:15:59.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:15:59.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:15:59.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:15:59.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:316: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:15:59.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:322: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:15:59.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:15:59.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:15:59.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:15:59.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:15:59.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-02-05T06:15:59.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:15:59.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:15:59.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:15:59.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:15:59.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-02-05T06:15:59.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-02-05T06:15:59.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:15:59.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:15:59.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:15:59.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:15:59.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:15:59.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:15:59.182 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:15:59.182 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:15:59.183 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:15:59.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:59.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:15:59.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:15:59.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:15:59.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:15:59.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:15:59.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:15:59.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:15:59.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:15:59.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:15:59.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:15:59.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:15:59.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2024-02-05T06:15:59.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-02-05T06:15:59.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:15:59.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a297fb34-6d47-48c7-9a9a-8595435d4b2e 2024-02-05T06:15:59.186 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 a297fb34-6d47-48c7-9a9a-8595435d4b2e 2024-02-05T06:15:59.187 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a297fb34-6d47-48c7-9a9a-8595435d4b2e' 2024-02-05T06:15:59.187 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:15:59.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAffcBlfqIkDRAAFAopNButa9oPZpM/iCAguQ== 2024-02-05T06:15:59.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAffcBlfqIkDRAAFAopNButa9oPZpM/iCAguQ=="}' 2024-02-05T06:15:59.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a297fb34-6d47-48c7-9a9a-8595435d4b2e -i td/osd-scrub-test/2/new.json 2024-02-05T06:15:59.535 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:15:59.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-02-05T06:15:59.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAffcBlfqIkDRAAFAopNButa9oPZpM/iCAguQ== --osd-uuid a297fb34-6d47-48c7-9a9a-8595435d4b2e 2024-02-05T06:15:59.576 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:59.607+0000 7f468e0184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:59.620 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:59.650+0000 7f468e0184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:59.652+0000 7f468e0184c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:15:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:59.652+0000 7f468e0184c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:15:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:59.652+0000 7f468e0184c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:15:59.621 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:59.652+0000 7f468e0184c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:15:59.694 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:15:59.725+0000 7f468e0184c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-02-05T06:16:05.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-02-05T06:16:05.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:16:05.694 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T06:16:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:16:05.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:16:06.036 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T06:16:06.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:16:06.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2024-02-05T06:16:06.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:16:06.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:16:06.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:16:06.063 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:16:06.094+0000 7fb4647fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:06.065 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:16:06.096+0000 7fb4647fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:06.066 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:16:06.097+0000 7fb4647fe4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:16:06.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:16:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:16:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:16:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:16:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:16:06.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:06.303 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:16:06.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:16:06.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:06.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:16:06.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:06.592 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:16:06.623+0000 7fb4647fe4c0 -1 Falling back to public interface 2024-02-05T06:16:07.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:07.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:07.570 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:16:07.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:16:07.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:07.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:16:07.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:08.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:08.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:08.856 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:16:08.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:16:08.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:08.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:16:08.911 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:16:08.942+0000 7fb4647fe4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T06:16:09.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:16:10.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:16:10.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-02-05T06:16:10.126 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:16:10.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:16:10.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:16:10.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:16:10.399 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3675941740,v1:127.0.0.1:6819/3675941740] [v2:127.0.0.1:6820/3675941740,v1:127.0.0.1:6821/3675941740] exists,up a297fb34-6d47-48c7-9a9a-8595435d4b2e 2024-02-05T06:16:10.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:16:10.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:16:10.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:16:10.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: create_pool test 1 1 2024-02-05T06:16:10.400 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:16:10.791 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T06:16:10.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:16:11.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:331: _scrub_abort: wait_for_clean 2024-02-05T06:16:11.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:16:11.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:16:11.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:16:11.804 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:16:11.805 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:16:11.805 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:16:11.805 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:16:11.805 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:16:11.805 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:16:11.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:16:11.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:16:11.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:16:11.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:16:11.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:16:11.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:16:12.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:16:12.184 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:16:12.184 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:16:12.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:16:12.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:16:12.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:16:12.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-02-05T06:16:12.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-02-05T06:16:12.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-02-05T06:16:12.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:16:12.285 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:16:12.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T06:16:12.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T06:16:12.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964' 2024-02-05T06:16:12.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:16:12.402 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:16:12.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-02-05T06:16:12.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-02-05T06:16:12.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964 2-64424509442' 2024-02-05T06:16:12.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:16:12.502 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-02-05T06:16:12.503 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:16:12.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:16:12.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-02-05T06:16:12.504 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:16:12.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-02-05T06:16:12.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-02-05T06:16:12.505 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836486 2024-02-05T06:16:12.505 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:16:12.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-02-05T06:16:12.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:16:13.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:16:13.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:16:14.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836486 2024-02-05T06:16:14.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:16:14.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T06:16:14.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:16:14.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:16:14.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T06:16:14.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:16:14.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T06:16:14.041 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672964 2024-02-05T06:16:14.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T06:16:14.042 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:16:14.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-02-05T06:16:14.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:16:14.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-02-05T06:16:14.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:16:14.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:16:14.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-02-05T06:16:14.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:16:14.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-02-05T06:16:14.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-02-05T06:16:14.309 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509442 2024-02-05T06:16:14.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:16:14.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-02-05T06:16:14.575 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:16:14.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:16:14.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:16:14.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:16:14.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:16:14.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:16:14.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:16:14.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:16:14.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:16:14.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:16:14.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:16:15.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:16:15.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:16:15.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:16:15.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:16:15.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:16:15.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T06:16:15.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:16:15.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T06:16:15.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:16:15.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:16:15.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:16:15.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:16:15.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:16:15.601 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:16:15.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:16:15.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:16:15.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:16:15.602 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:16:15.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:16:15.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:16:15.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:16:15.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:16:16.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:16:16.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:16:16.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:16:16.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: ceph osd dump 2024-02-05T06:16:16.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: grep '^pool.*['\'']test['\'']' 2024-02-05T06:16:16.194 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: awk '{ print $2 }' 2024-02-05T06:16:16.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: poolid=1 2024-02-05T06:16:16.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: dd if=/dev/urandom of=testdata.548953 bs=1032 count=1 2024-02-05T06:16:16.457 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:16:16.457 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:16:16.457 INFO:tasks.workunit.client.0.smithi188.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.6972e-05 s, 22.0 MB/s 2024-02-05T06:16:16.457 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: seq 1 1000 2024-02-05T06:16:16.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:16.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj1 testdata.548953 2024-02-05T06:16:16.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:16.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj2 testdata.548953 2024-02-05T06:16:16.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:16.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj3 testdata.548953 2024-02-05T06:16:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj4 testdata.548953 2024-02-05T06:16:17.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:17.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj5 testdata.548953 2024-02-05T06:16:17.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:17.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj6 testdata.548953 2024-02-05T06:16:17.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:17.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj7 testdata.548953 2024-02-05T06:16:17.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:17.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj8 testdata.548953 2024-02-05T06:16:17.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:17.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj9 testdata.548953 2024-02-05T06:16:17.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:17.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj10 testdata.548953 2024-02-05T06:16:17.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:17.962 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj11 testdata.548953 2024-02-05T06:16:18.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:18.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj12 testdata.548953 2024-02-05T06:16:18.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:18.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj13 testdata.548953 2024-02-05T06:16:18.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:18.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj14 testdata.548953 2024-02-05T06:16:18.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:18.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj15 testdata.548953 2024-02-05T06:16:18.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:18.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj16 testdata.548953 2024-02-05T06:16:18.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:18.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj17 testdata.548953 2024-02-05T06:16:18.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:18.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj18 testdata.548953 2024-02-05T06:16:19.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:19.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj19 testdata.548953 2024-02-05T06:16:19.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:19.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj20 testdata.548953 2024-02-05T06:16:19.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:19.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj21 testdata.548953 2024-02-05T06:16:19.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:19.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj22 testdata.548953 2024-02-05T06:16:19.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:19.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj23 testdata.548953 2024-02-05T06:16:19.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:19.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj24 testdata.548953 2024-02-05T06:16:19.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:19.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj25 testdata.548953 2024-02-05T06:16:20.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:20.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj26 testdata.548953 2024-02-05T06:16:20.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:20.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj27 testdata.548953 2024-02-05T06:16:20.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:20.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj28 testdata.548953 2024-02-05T06:16:20.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:20.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj29 testdata.548953 2024-02-05T06:16:20.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:20.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj30 testdata.548953 2024-02-05T06:16:20.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:20.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj31 testdata.548953 2024-02-05T06:16:20.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:20.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj32 testdata.548953 2024-02-05T06:16:21.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:21.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj33 testdata.548953 2024-02-05T06:16:21.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:21.206 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj34 testdata.548953 2024-02-05T06:16:21.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:21.340 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj35 testdata.548953 2024-02-05T06:16:21.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:21.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj36 testdata.548953 2024-02-05T06:16:21.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:21.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj37 testdata.548953 2024-02-05T06:16:21.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:21.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj38 testdata.548953 2024-02-05T06:16:21.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:21.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj39 testdata.548953 2024-02-05T06:16:22.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:22.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj40 testdata.548953 2024-02-05T06:16:22.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:22.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj41 testdata.548953 2024-02-05T06:16:22.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:22.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj42 testdata.548953 2024-02-05T06:16:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:22.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj43 testdata.548953 2024-02-05T06:16:22.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:22.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj44 testdata.548953 2024-02-05T06:16:22.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:22.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj45 testdata.548953 2024-02-05T06:16:22.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:22.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj46 testdata.548953 2024-02-05T06:16:23.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:23.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj47 testdata.548953 2024-02-05T06:16:23.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:23.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj48 testdata.548953 2024-02-05T06:16:23.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:23.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj49 testdata.548953 2024-02-05T06:16:23.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:23.430 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj50 testdata.548953 2024-02-05T06:16:23.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:23.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj51 testdata.548953 2024-02-05T06:16:23.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:23.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj52 testdata.548953 2024-02-05T06:16:23.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:23.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj53 testdata.548953 2024-02-05T06:16:24.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj54 testdata.548953 2024-02-05T06:16:24.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj55 testdata.548953 2024-02-05T06:16:24.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj56 testdata.548953 2024-02-05T06:16:24.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj57 testdata.548953 2024-02-05T06:16:24.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj58 testdata.548953 2024-02-05T06:16:24.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj59 testdata.548953 2024-02-05T06:16:24.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj60 testdata.548953 2024-02-05T06:16:24.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:24.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj61 testdata.548953 2024-02-05T06:16:25.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:25.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj62 testdata.548953 2024-02-05T06:16:25.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:25.216 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj63 testdata.548953 2024-02-05T06:16:25.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:25.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj64 testdata.548953 2024-02-05T06:16:25.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:25.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj65 testdata.548953 2024-02-05T06:16:25.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:25.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj66 testdata.548953 2024-02-05T06:16:26.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:26.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj67 testdata.548953 2024-02-05T06:16:26.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:26.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj68 testdata.548953 2024-02-05T06:16:26.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:26.355 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj69 testdata.548953 2024-02-05T06:16:26.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:26.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj70 testdata.548953 2024-02-05T06:16:26.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:26.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj71 testdata.548953 2024-02-05T06:16:26.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:26.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj72 testdata.548953 2024-02-05T06:16:26.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:26.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj73 testdata.548953 2024-02-05T06:16:27.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:27.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj74 testdata.548953 2024-02-05T06:16:27.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:27.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj75 testdata.548953 2024-02-05T06:16:27.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:27.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj76 testdata.548953 2024-02-05T06:16:27.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:27.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj77 testdata.548953 2024-02-05T06:16:27.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:27.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj78 testdata.548953 2024-02-05T06:16:27.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:27.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj79 testdata.548953 2024-02-05T06:16:27.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:27.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj80 testdata.548953 2024-02-05T06:16:28.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj81 testdata.548953 2024-02-05T06:16:28.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj82 testdata.548953 2024-02-05T06:16:28.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.268 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj83 testdata.548953 2024-02-05T06:16:28.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj84 testdata.548953 2024-02-05T06:16:28.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj85 testdata.548953 2024-02-05T06:16:28.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj86 testdata.548953 2024-02-05T06:16:28.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj87 testdata.548953 2024-02-05T06:16:28.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:28.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj88 testdata.548953 2024-02-05T06:16:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:29.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj89 testdata.548953 2024-02-05T06:16:29.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:29.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj90 testdata.548953 2024-02-05T06:16:29.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:29.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj91 testdata.548953 2024-02-05T06:16:29.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:29.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj92 testdata.548953 2024-02-05T06:16:29.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:29.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj93 testdata.548953 2024-02-05T06:16:29.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:29.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj94 testdata.548953 2024-02-05T06:16:30.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj95 testdata.548953 2024-02-05T06:16:30.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj96 testdata.548953 2024-02-05T06:16:30.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj97 testdata.548953 2024-02-05T06:16:30.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj98 testdata.548953 2024-02-05T06:16:30.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj99 testdata.548953 2024-02-05T06:16:30.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj100 testdata.548953 2024-02-05T06:16:30.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj101 testdata.548953 2024-02-05T06:16:30.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:30.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj102 testdata.548953 2024-02-05T06:16:31.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:31.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj103 testdata.548953 2024-02-05T06:16:31.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:31.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj104 testdata.548953 2024-02-05T06:16:31.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:31.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj105 testdata.548953 2024-02-05T06:16:31.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:31.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj106 testdata.548953 2024-02-05T06:16:31.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:31.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj107 testdata.548953 2024-02-05T06:16:31.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:31.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj108 testdata.548953 2024-02-05T06:16:31.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:31.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj109 testdata.548953 2024-02-05T06:16:32.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:32.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj110 testdata.548953 2024-02-05T06:16:32.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:32.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj111 testdata.548953 2024-02-05T06:16:32.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:32.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj112 testdata.548953 2024-02-05T06:16:32.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:32.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj113 testdata.548953 2024-02-05T06:16:32.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:32.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj114 testdata.548953 2024-02-05T06:16:32.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:32.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj115 testdata.548953 2024-02-05T06:16:32.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:32.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj116 testdata.548953 2024-02-05T06:16:33.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:33.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj117 testdata.548953 2024-02-05T06:16:33.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:33.166 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj118 testdata.548953 2024-02-05T06:16:33.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:33.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj119 testdata.548953 2024-02-05T06:16:33.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:33.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj120 testdata.548953 2024-02-05T06:16:33.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:33.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj121 testdata.548953 2024-02-05T06:16:33.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:33.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj122 testdata.548953 2024-02-05T06:16:33.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:33.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj123 testdata.548953 2024-02-05T06:16:34.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj124 testdata.548953 2024-02-05T06:16:34.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj125 testdata.548953 2024-02-05T06:16:34.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj126 testdata.548953 2024-02-05T06:16:34.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj127 testdata.548953 2024-02-05T06:16:34.540 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj128 testdata.548953 2024-02-05T06:16:34.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj129 testdata.548953 2024-02-05T06:16:34.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj130 testdata.548953 2024-02-05T06:16:34.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:34.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj131 testdata.548953 2024-02-05T06:16:35.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:35.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj132 testdata.548953 2024-02-05T06:16:35.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:35.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj133 testdata.548953 2024-02-05T06:16:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:35.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj134 testdata.548953 2024-02-05T06:16:35.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:35.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj135 testdata.548953 2024-02-05T06:16:35.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:35.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj136 testdata.548953 2024-02-05T06:16:35.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:35.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj137 testdata.548953 2024-02-05T06:16:35.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:35.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj138 testdata.548953 2024-02-05T06:16:36.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj139 testdata.548953 2024-02-05T06:16:36.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj140 testdata.548953 2024-02-05T06:16:36.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj141 testdata.548953 2024-02-05T06:16:36.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj142 testdata.548953 2024-02-05T06:16:36.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj143 testdata.548953 2024-02-05T06:16:36.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj144 testdata.548953 2024-02-05T06:16:36.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj145 testdata.548953 2024-02-05T06:16:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:36.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj146 testdata.548953 2024-02-05T06:16:37.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:37.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj147 testdata.548953 2024-02-05T06:16:37.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:37.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj148 testdata.548953 2024-02-05T06:16:37.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:37.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj149 testdata.548953 2024-02-05T06:16:37.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:37.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj150 testdata.548953 2024-02-05T06:16:37.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:37.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj151 testdata.548953 2024-02-05T06:16:37.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:37.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj152 testdata.548953 2024-02-05T06:16:37.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:37.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj153 testdata.548953 2024-02-05T06:16:38.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:38.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj154 testdata.548953 2024-02-05T06:16:38.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:38.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj155 testdata.548953 2024-02-05T06:16:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:38.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj156 testdata.548953 2024-02-05T06:16:38.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:38.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj157 testdata.548953 2024-02-05T06:16:38.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:38.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj158 testdata.548953 2024-02-05T06:16:38.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:38.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj159 testdata.548953 2024-02-05T06:16:38.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:38.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj160 testdata.548953 2024-02-05T06:16:39.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:39.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj161 testdata.548953 2024-02-05T06:16:39.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:39.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj162 testdata.548953 2024-02-05T06:16:39.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:39.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj163 testdata.548953 2024-02-05T06:16:39.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:39.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj164 testdata.548953 2024-02-05T06:16:39.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:39.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj165 testdata.548953 2024-02-05T06:16:39.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:39.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj166 testdata.548953 2024-02-05T06:16:39.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:39.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj167 testdata.548953 2024-02-05T06:16:40.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj168 testdata.548953 2024-02-05T06:16:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj169 testdata.548953 2024-02-05T06:16:40.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj170 testdata.548953 2024-02-05T06:16:40.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj171 testdata.548953 2024-02-05T06:16:40.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj172 testdata.548953 2024-02-05T06:16:40.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj173 testdata.548953 2024-02-05T06:16:40.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj174 testdata.548953 2024-02-05T06:16:40.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:40.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj175 testdata.548953 2024-02-05T06:16:41.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:41.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj176 testdata.548953 2024-02-05T06:16:41.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:41.255 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj177 testdata.548953 2024-02-05T06:16:41.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:41.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj178 testdata.548953 2024-02-05T06:16:41.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:41.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj179 testdata.548953 2024-02-05T06:16:41.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:41.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj180 testdata.548953 2024-02-05T06:16:41.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:41.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj181 testdata.548953 2024-02-05T06:16:41.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:41.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj182 testdata.548953 2024-02-05T06:16:42.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:42.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj183 testdata.548953 2024-02-05T06:16:42.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:42.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj184 testdata.548953 2024-02-05T06:16:42.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:42.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj185 testdata.548953 2024-02-05T06:16:42.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:42.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj186 testdata.548953 2024-02-05T06:16:42.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:42.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj187 testdata.548953 2024-02-05T06:16:42.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:42.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj188 testdata.548953 2024-02-05T06:16:42.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:42.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj189 testdata.548953 2024-02-05T06:16:43.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:43.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj190 testdata.548953 2024-02-05T06:16:43.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:43.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj191 testdata.548953 2024-02-05T06:16:43.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:43.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj192 testdata.548953 2024-02-05T06:16:43.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:43.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj193 testdata.548953 2024-02-05T06:16:43.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:43.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj194 testdata.548953 2024-02-05T06:16:43.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:43.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj195 testdata.548953 2024-02-05T06:16:43.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:43.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj196 testdata.548953 2024-02-05T06:16:44.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj197 testdata.548953 2024-02-05T06:16:44.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj198 testdata.548953 2024-02-05T06:16:44.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj199 testdata.548953 2024-02-05T06:16:44.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj200 testdata.548953 2024-02-05T06:16:44.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj201 testdata.548953 2024-02-05T06:16:44.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj202 testdata.548953 2024-02-05T06:16:44.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj203 testdata.548953 2024-02-05T06:16:44.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:44.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj204 testdata.548953 2024-02-05T06:16:45.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:45.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj205 testdata.548953 2024-02-05T06:16:45.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:45.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj206 testdata.548953 2024-02-05T06:16:45.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:45.329 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj207 testdata.548953 2024-02-05T06:16:45.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:45.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj208 testdata.548953 2024-02-05T06:16:45.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:45.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj209 testdata.548953 2024-02-05T06:16:45.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:45.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj210 testdata.548953 2024-02-05T06:16:45.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:45.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj211 testdata.548953 2024-02-05T06:16:46.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj212 testdata.548953 2024-02-05T06:16:46.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj213 testdata.548953 2024-02-05T06:16:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj214 testdata.548953 2024-02-05T06:16:46.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj215 testdata.548953 2024-02-05T06:16:46.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj216 testdata.548953 2024-02-05T06:16:46.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.662 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj217 testdata.548953 2024-02-05T06:16:46.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj218 testdata.548953 2024-02-05T06:16:46.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:46.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj219 testdata.548953 2024-02-05T06:16:47.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:47.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj220 testdata.548953 2024-02-05T06:16:47.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:47.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj221 testdata.548953 2024-02-05T06:16:47.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:47.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj222 testdata.548953 2024-02-05T06:16:47.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:47.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj223 testdata.548953 2024-02-05T06:16:47.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:47.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj224 testdata.548953 2024-02-05T06:16:47.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:47.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj225 testdata.548953 2024-02-05T06:16:47.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:47.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj226 testdata.548953 2024-02-05T06:16:48.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj227 testdata.548953 2024-02-05T06:16:48.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj228 testdata.548953 2024-02-05T06:16:48.280 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj229 testdata.548953 2024-02-05T06:16:48.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj230 testdata.548953 2024-02-05T06:16:48.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj231 testdata.548953 2024-02-05T06:16:48.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj232 testdata.548953 2024-02-05T06:16:48.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj233 testdata.548953 2024-02-05T06:16:48.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:48.979 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj234 testdata.548953 2024-02-05T06:16:49.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:49.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj235 testdata.548953 2024-02-05T06:16:49.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:49.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj236 testdata.548953 2024-02-05T06:16:49.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:49.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj237 testdata.548953 2024-02-05T06:16:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:49.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj238 testdata.548953 2024-02-05T06:16:49.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:49.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj239 testdata.548953 2024-02-05T06:16:49.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:49.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj240 testdata.548953 2024-02-05T06:16:49.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:49.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj241 testdata.548953 2024-02-05T06:16:50.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:50.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj242 testdata.548953 2024-02-05T06:16:50.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:50.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj243 testdata.548953 2024-02-05T06:16:50.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:50.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj244 testdata.548953 2024-02-05T06:16:50.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:50.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj245 testdata.548953 2024-02-05T06:16:50.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:50.619 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj246 testdata.548953 2024-02-05T06:16:50.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:50.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj247 testdata.548953 2024-02-05T06:16:50.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:50.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj248 testdata.548953 2024-02-05T06:16:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:51.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj249 testdata.548953 2024-02-05T06:16:51.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:51.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj250 testdata.548953 2024-02-05T06:16:51.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:51.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj251 testdata.548953 2024-02-05T06:16:51.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:51.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj252 testdata.548953 2024-02-05T06:16:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj253 testdata.548953 2024-02-05T06:16:51.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:51.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj254 testdata.548953 2024-02-05T06:16:51.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:51.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj255 testdata.548953 2024-02-05T06:16:52.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj256 testdata.548953 2024-02-05T06:16:52.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj257 testdata.548953 2024-02-05T06:16:52.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj258 testdata.548953 2024-02-05T06:16:52.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj259 testdata.548953 2024-02-05T06:16:52.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj260 testdata.548953 2024-02-05T06:16:52.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj261 testdata.548953 2024-02-05T06:16:52.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj262 testdata.548953 2024-02-05T06:16:52.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:52.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj263 testdata.548953 2024-02-05T06:16:53.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:53.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj264 testdata.548953 2024-02-05T06:16:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:53.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj265 testdata.548953 2024-02-05T06:16:53.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:53.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj266 testdata.548953 2024-02-05T06:16:53.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:53.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj267 testdata.548953 2024-02-05T06:16:53.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:53.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj268 testdata.548953 2024-02-05T06:16:53.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:53.789 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj269 testdata.548953 2024-02-05T06:16:53.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:53.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj270 testdata.548953 2024-02-05T06:16:54.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:54.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj271 testdata.548953 2024-02-05T06:16:54.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:54.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj272 testdata.548953 2024-02-05T06:16:54.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:54.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj273 testdata.548953 2024-02-05T06:16:54.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:54.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj274 testdata.548953 2024-02-05T06:16:54.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:54.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj275 testdata.548953 2024-02-05T06:16:54.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:54.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj276 testdata.548953 2024-02-05T06:16:54.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:54.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj277 testdata.548953 2024-02-05T06:16:55.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.013 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj278 testdata.548953 2024-02-05T06:16:55.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj279 testdata.548953 2024-02-05T06:16:55.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj280 testdata.548953 2024-02-05T06:16:55.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj281 testdata.548953 2024-02-05T06:16:55.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj282 testdata.548953 2024-02-05T06:16:55.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj283 testdata.548953 2024-02-05T06:16:55.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj284 testdata.548953 2024-02-05T06:16:55.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:55.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj285 testdata.548953 2024-02-05T06:16:56.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:56.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj286 testdata.548953 2024-02-05T06:16:56.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:56.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj287 testdata.548953 2024-02-05T06:16:56.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:56.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj288 testdata.548953 2024-02-05T06:16:56.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:56.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj289 testdata.548953 2024-02-05T06:16:56.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:56.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj290 testdata.548953 2024-02-05T06:16:56.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:56.773 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj291 testdata.548953 2024-02-05T06:16:56.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:56.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj292 testdata.548953 2024-02-05T06:16:57.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:57.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj293 testdata.548953 2024-02-05T06:16:57.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:57.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj294 testdata.548953 2024-02-05T06:16:57.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:57.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj295 testdata.548953 2024-02-05T06:16:57.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:57.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj296 testdata.548953 2024-02-05T06:16:58.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:58.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj297 testdata.548953 2024-02-05T06:16:58.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:58.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj298 testdata.548953 2024-02-05T06:16:58.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:58.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj299 testdata.548953 2024-02-05T06:16:58.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:58.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj300 testdata.548953 2024-02-05T06:16:58.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:58.649 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj301 testdata.548953 2024-02-05T06:16:58.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:58.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj302 testdata.548953 2024-02-05T06:16:58.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:58.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj303 testdata.548953 2024-02-05T06:16:59.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:59.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj304 testdata.548953 2024-02-05T06:16:59.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:59.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj305 testdata.548953 2024-02-05T06:16:59.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:59.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj306 testdata.548953 2024-02-05T06:16:59.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:59.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj307 testdata.548953 2024-02-05T06:16:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:59.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj308 testdata.548953 2024-02-05T06:16:59.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:59.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj309 testdata.548953 2024-02-05T06:16:59.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:16:59.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj310 testdata.548953 2024-02-05T06:17:00.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:00.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj311 testdata.548953 2024-02-05T06:17:00.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:00.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj312 testdata.548953 2024-02-05T06:17:00.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:00.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj313 testdata.548953 2024-02-05T06:17:00.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:00.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj314 testdata.548953 2024-02-05T06:17:00.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:00.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj315 testdata.548953 2024-02-05T06:17:00.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:00.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj316 testdata.548953 2024-02-05T06:17:01.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:01.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj317 testdata.548953 2024-02-05T06:17:01.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:01.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj318 testdata.548953 2024-02-05T06:17:01.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:01.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj319 testdata.548953 2024-02-05T06:17:01.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:01.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj320 testdata.548953 2024-02-05T06:17:01.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:01.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj321 testdata.548953 2024-02-05T06:17:01.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:01.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj322 testdata.548953 2024-02-05T06:17:01.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:01.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj323 testdata.548953 2024-02-05T06:17:02.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:02.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj324 testdata.548953 2024-02-05T06:17:02.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:02.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj325 testdata.548953 2024-02-05T06:17:02.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:02.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj326 testdata.548953 2024-02-05T06:17:02.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:02.487 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj327 testdata.548953 2024-02-05T06:17:02.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:02.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj328 testdata.548953 2024-02-05T06:17:02.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:02.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj329 testdata.548953 2024-02-05T06:17:02.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:02.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj330 testdata.548953 2024-02-05T06:17:03.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:03.065 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj331 testdata.548953 2024-02-05T06:17:03.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:03.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj332 testdata.548953 2024-02-05T06:17:03.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:03.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj333 testdata.548953 2024-02-05T06:17:03.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:03.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj334 testdata.548953 2024-02-05T06:17:03.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:03.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj335 testdata.548953 2024-02-05T06:17:03.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:03.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj336 testdata.548953 2024-02-05T06:17:03.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:03.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj337 testdata.548953 2024-02-05T06:17:04.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:04.046 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj338 testdata.548953 2024-02-05T06:17:04.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:04.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj339 testdata.548953 2024-02-05T06:17:04.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:04.339 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj340 testdata.548953 2024-02-05T06:17:04.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:04.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj341 testdata.548953 2024-02-05T06:17:04.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:04.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj342 testdata.548953 2024-02-05T06:17:04.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:04.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj343 testdata.548953 2024-02-05T06:17:04.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:04.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj344 testdata.548953 2024-02-05T06:17:05.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:05.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj345 testdata.548953 2024-02-05T06:17:05.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:05.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj346 testdata.548953 2024-02-05T06:17:05.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:05.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj347 testdata.548953 2024-02-05T06:17:05.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:05.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj348 testdata.548953 2024-02-05T06:17:05.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:05.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj349 testdata.548953 2024-02-05T06:17:05.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:05.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj350 testdata.548953 2024-02-05T06:17:05.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:05.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj351 testdata.548953 2024-02-05T06:17:06.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj352 testdata.548953 2024-02-05T06:17:06.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj353 testdata.548953 2024-02-05T06:17:06.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj354 testdata.548953 2024-02-05T06:17:06.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.418 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj355 testdata.548953 2024-02-05T06:17:06.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj356 testdata.548953 2024-02-05T06:17:06.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj357 testdata.548953 2024-02-05T06:17:06.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj358 testdata.548953 2024-02-05T06:17:06.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:06.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj359 testdata.548953 2024-02-05T06:17:07.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:07.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj360 testdata.548953 2024-02-05T06:17:07.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:07.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj361 testdata.548953 2024-02-05T06:17:07.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:07.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj362 testdata.548953 2024-02-05T06:17:07.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:07.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj363 testdata.548953 2024-02-05T06:17:07.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:07.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj364 testdata.548953 2024-02-05T06:17:07.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:07.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj365 testdata.548953 2024-02-05T06:17:08.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj366 testdata.548953 2024-02-05T06:17:08.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj367 testdata.548953 2024-02-05T06:17:08.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj368 testdata.548953 2024-02-05T06:17:08.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj369 testdata.548953 2024-02-05T06:17:08.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj370 testdata.548953 2024-02-05T06:17:08.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj371 testdata.548953 2024-02-05T06:17:08.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj372 testdata.548953 2024-02-05T06:17:08.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:08.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj373 testdata.548953 2024-02-05T06:17:09.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:09.092 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj374 testdata.548953 2024-02-05T06:17:09.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:09.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj375 testdata.548953 2024-02-05T06:17:09.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:09.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj376 testdata.548953 2024-02-05T06:17:09.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:09.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj377 testdata.548953 2024-02-05T06:17:09.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:09.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj378 testdata.548953 2024-02-05T06:17:09.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:09.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj379 testdata.548953 2024-02-05T06:17:09.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:09.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj380 testdata.548953 2024-02-05T06:17:10.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:10.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj381 testdata.548953 2024-02-05T06:17:10.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:10.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj382 testdata.548953 2024-02-05T06:17:10.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:10.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj383 testdata.548953 2024-02-05T06:17:10.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:10.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj384 testdata.548953 2024-02-05T06:17:10.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:10.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj385 testdata.548953 2024-02-05T06:17:10.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:10.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj386 testdata.548953 2024-02-05T06:17:10.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:10.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj387 testdata.548953 2024-02-05T06:17:11.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:11.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj388 testdata.548953 2024-02-05T06:17:11.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:11.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj389 testdata.548953 2024-02-05T06:17:11.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:11.330 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj390 testdata.548953 2024-02-05T06:17:11.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:11.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj391 testdata.548953 2024-02-05T06:17:11.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:11.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj392 testdata.548953 2024-02-05T06:17:11.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:11.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj393 testdata.548953 2024-02-05T06:17:11.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:11.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj394 testdata.548953 2024-02-05T06:17:12.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:12.043 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj395 testdata.548953 2024-02-05T06:17:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj396 testdata.548953 2024-02-05T06:17:12.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:12.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj397 testdata.548953 2024-02-05T06:17:12.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:12.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj398 testdata.548953 2024-02-05T06:17:12.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:12.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj399 testdata.548953 2024-02-05T06:17:12.738 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:12.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj400 testdata.548953 2024-02-05T06:17:12.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:12.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj401 testdata.548953 2024-02-05T06:17:13.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:13.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj402 testdata.548953 2024-02-05T06:17:13.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:13.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj403 testdata.548953 2024-02-05T06:17:13.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:13.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj404 testdata.548953 2024-02-05T06:17:13.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:13.417 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj405 testdata.548953 2024-02-05T06:17:13.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:13.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj406 testdata.548953 2024-02-05T06:17:13.702 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:13.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj407 testdata.548953 2024-02-05T06:17:13.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:13.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj408 testdata.548953 2024-02-05T06:17:14.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj409 testdata.548953 2024-02-05T06:17:14.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj410 testdata.548953 2024-02-05T06:17:14.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj411 testdata.548953 2024-02-05T06:17:14.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj412 testdata.548953 2024-02-05T06:17:14.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj413 testdata.548953 2024-02-05T06:17:14.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj414 testdata.548953 2024-02-05T06:17:14.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj415 testdata.548953 2024-02-05T06:17:14.976 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:14.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj416 testdata.548953 2024-02-05T06:17:15.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:15.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj417 testdata.548953 2024-02-05T06:17:15.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:15.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj418 testdata.548953 2024-02-05T06:17:15.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:15.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj419 testdata.548953 2024-02-05T06:17:15.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:15.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj420 testdata.548953 2024-02-05T06:17:15.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:15.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj421 testdata.548953 2024-02-05T06:17:15.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:15.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj422 testdata.548953 2024-02-05T06:17:15.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:15.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj423 testdata.548953 2024-02-05T06:17:16.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:16.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj424 testdata.548953 2024-02-05T06:17:16.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:16.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj425 testdata.548953 2024-02-05T06:17:16.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:16.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj426 testdata.548953 2024-02-05T06:17:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:16.539 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj427 testdata.548953 2024-02-05T06:17:16.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:16.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj428 testdata.548953 2024-02-05T06:17:16.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:16.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj429 testdata.548953 2024-02-05T06:17:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:16.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj430 testdata.548953 2024-02-05T06:17:17.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:17.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj431 testdata.548953 2024-02-05T06:17:17.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:17.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj432 testdata.548953 2024-02-05T06:17:17.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:17.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj433 testdata.548953 2024-02-05T06:17:17.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:17.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj434 testdata.548953 2024-02-05T06:17:17.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:17.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj435 testdata.548953 2024-02-05T06:17:17.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:17.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj436 testdata.548953 2024-02-05T06:17:17.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:17.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj437 testdata.548953 2024-02-05T06:17:18.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj438 testdata.548953 2024-02-05T06:17:18.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj439 testdata.548953 2024-02-05T06:17:18.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj440 testdata.548953 2024-02-05T06:17:18.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj441 testdata.548953 2024-02-05T06:17:18.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj442 testdata.548953 2024-02-05T06:17:18.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.718 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj443 testdata.548953 2024-02-05T06:17:18.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj444 testdata.548953 2024-02-05T06:17:18.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:18.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj445 testdata.548953 2024-02-05T06:17:19.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:19.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj446 testdata.548953 2024-02-05T06:17:19.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:19.264 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj447 testdata.548953 2024-02-05T06:17:19.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:19.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj448 testdata.548953 2024-02-05T06:17:19.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:19.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj449 testdata.548953 2024-02-05T06:17:19.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:19.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj450 testdata.548953 2024-02-05T06:17:19.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:19.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj451 testdata.548953 2024-02-05T06:17:20.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj452 testdata.548953 2024-02-05T06:17:20.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj453 testdata.548953 2024-02-05T06:17:20.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj454 testdata.548953 2024-02-05T06:17:20.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj455 testdata.548953 2024-02-05T06:17:20.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj456 testdata.548953 2024-02-05T06:17:20.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj457 testdata.548953 2024-02-05T06:17:20.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj458 testdata.548953 2024-02-05T06:17:20.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:20.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj459 testdata.548953 2024-02-05T06:17:21.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:21.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj460 testdata.548953 2024-02-05T06:17:21.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:21.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj461 testdata.548953 2024-02-05T06:17:21.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:21.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj462 testdata.548953 2024-02-05T06:17:21.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:21.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj463 testdata.548953 2024-02-05T06:17:21.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:21.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj464 testdata.548953 2024-02-05T06:17:21.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:21.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj465 testdata.548953 2024-02-05T06:17:21.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:21.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj466 testdata.548953 2024-02-05T06:17:22.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:22.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj467 testdata.548953 2024-02-05T06:17:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj468 testdata.548953 2024-02-05T06:17:22.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:22.391 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj469 testdata.548953 2024-02-05T06:17:22.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:22.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj470 testdata.548953 2024-02-05T06:17:22.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:22.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj471 testdata.548953 2024-02-05T06:17:22.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:22.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj472 testdata.548953 2024-02-05T06:17:22.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:22.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj473 testdata.548953 2024-02-05T06:17:23.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:23.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj474 testdata.548953 2024-02-05T06:17:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:23.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj475 testdata.548953 2024-02-05T06:17:23.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:23.356 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj476 testdata.548953 2024-02-05T06:17:23.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:23.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj477 testdata.548953 2024-02-05T06:17:23.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:23.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj478 testdata.548953 2024-02-05T06:17:23.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:23.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj479 testdata.548953 2024-02-05T06:17:23.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:23.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj480 testdata.548953 2024-02-05T06:17:24.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:24.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj481 testdata.548953 2024-02-05T06:17:24.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:24.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj482 testdata.548953 2024-02-05T06:17:24.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:24.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj483 testdata.548953 2024-02-05T06:17:24.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:24.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj484 testdata.548953 2024-02-05T06:17:24.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:24.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj485 testdata.548953 2024-02-05T06:17:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj486 testdata.548953 2024-02-05T06:17:24.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:24.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj487 testdata.548953 2024-02-05T06:17:25.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:25.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj488 testdata.548953 2024-02-05T06:17:25.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:25.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj489 testdata.548953 2024-02-05T06:17:25.392 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:25.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj490 testdata.548953 2024-02-05T06:17:25.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:25.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj491 testdata.548953 2024-02-05T06:17:25.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:25.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj492 testdata.548953 2024-02-05T06:17:25.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:25.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj493 testdata.548953 2024-02-05T06:17:26.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj494 testdata.548953 2024-02-05T06:17:26.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj495 testdata.548953 2024-02-05T06:17:26.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.274 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj496 testdata.548953 2024-02-05T06:17:26.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj497 testdata.548953 2024-02-05T06:17:26.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj498 testdata.548953 2024-02-05T06:17:26.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj499 testdata.548953 2024-02-05T06:17:26.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj500 testdata.548953 2024-02-05T06:17:26.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:26.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj501 testdata.548953 2024-02-05T06:17:27.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:27.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj502 testdata.548953 2024-02-05T06:17:27.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:27.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj503 testdata.548953 2024-02-05T06:17:27.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:27.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj504 testdata.548953 2024-02-05T06:17:27.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:27.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj505 testdata.548953 2024-02-05T06:17:27.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:27.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj506 testdata.548953 2024-02-05T06:17:27.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:27.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj507 testdata.548953 2024-02-05T06:17:27.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:27.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj508 testdata.548953 2024-02-05T06:17:28.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:28.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj509 testdata.548953 2024-02-05T06:17:28.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:28.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj510 testdata.548953 2024-02-05T06:17:28.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:28.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj511 testdata.548953 2024-02-05T06:17:28.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:28.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj512 testdata.548953 2024-02-05T06:17:29.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:29.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj513 testdata.548953 2024-02-05T06:17:29.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:29.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj514 testdata.548953 2024-02-05T06:17:29.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:29.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj515 testdata.548953 2024-02-05T06:17:29.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:29.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj516 testdata.548953 2024-02-05T06:17:29.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:29.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj517 testdata.548953 2024-02-05T06:17:29.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:29.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj518 testdata.548953 2024-02-05T06:17:29.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:29.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj519 testdata.548953 2024-02-05T06:17:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:30.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj520 testdata.548953 2024-02-05T06:17:30.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:30.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj521 testdata.548953 2024-02-05T06:17:30.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:30.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj522 testdata.548953 2024-02-05T06:17:30.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:30.489 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj523 testdata.548953 2024-02-05T06:17:30.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:30.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj524 testdata.548953 2024-02-05T06:17:30.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:30.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj525 testdata.548953 2024-02-05T06:17:30.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:30.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj526 testdata.548953 2024-02-05T06:17:31.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:31.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj527 testdata.548953 2024-02-05T06:17:31.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:31.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj528 testdata.548953 2024-02-05T06:17:31.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:31.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj529 testdata.548953 2024-02-05T06:17:31.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:31.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj530 testdata.548953 2024-02-05T06:17:31.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:31.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj531 testdata.548953 2024-02-05T06:17:31.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:31.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj532 testdata.548953 2024-02-05T06:17:32.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:32.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj533 testdata.548953 2024-02-05T06:17:32.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:32.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj534 testdata.548953 2024-02-05T06:17:32.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:32.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj535 testdata.548953 2024-02-05T06:17:32.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:32.434 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj536 testdata.548953 2024-02-05T06:17:32.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:32.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj537 testdata.548953 2024-02-05T06:17:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:32.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj538 testdata.548953 2024-02-05T06:17:32.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:32.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj539 testdata.548953 2024-02-05T06:17:33.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:33.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj540 testdata.548953 2024-02-05T06:17:33.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:33.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj541 testdata.548953 2024-02-05T06:17:33.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:33.323 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj542 testdata.548953 2024-02-05T06:17:33.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:33.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj543 testdata.548953 2024-02-05T06:17:33.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:33.624 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj544 testdata.548953 2024-02-05T06:17:33.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:33.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj545 testdata.548953 2024-02-05T06:17:33.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:33.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj546 testdata.548953 2024-02-05T06:17:34.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:34.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj547 testdata.548953 2024-02-05T06:17:34.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:34.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj548 testdata.548953 2024-02-05T06:17:34.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:34.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj549 testdata.548953 2024-02-05T06:17:34.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:34.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj550 testdata.548953 2024-02-05T06:17:34.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:34.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj551 testdata.548953 2024-02-05T06:17:34.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:34.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj552 testdata.548953 2024-02-05T06:17:34.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:34.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj553 testdata.548953 2024-02-05T06:17:35.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:35.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj554 testdata.548953 2024-02-05T06:17:35.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:35.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj555 testdata.548953 2024-02-05T06:17:35.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:35.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj556 testdata.548953 2024-02-05T06:17:35.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:35.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj557 testdata.548953 2024-02-05T06:17:35.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:35.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj558 testdata.548953 2024-02-05T06:17:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:35.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj559 testdata.548953 2024-02-05T06:17:36.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj560 testdata.548953 2024-02-05T06:17:36.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj561 testdata.548953 2024-02-05T06:17:36.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj562 testdata.548953 2024-02-05T06:17:36.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj563 testdata.548953 2024-02-05T06:17:36.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj564 testdata.548953 2024-02-05T06:17:36.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj565 testdata.548953 2024-02-05T06:17:36.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj566 testdata.548953 2024-02-05T06:17:36.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:36.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj567 testdata.548953 2024-02-05T06:17:37.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:37.136 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj568 testdata.548953 2024-02-05T06:17:37.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:37.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj569 testdata.548953 2024-02-05T06:17:37.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:37.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj570 testdata.548953 2024-02-05T06:17:37.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:37.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj571 testdata.548953 2024-02-05T06:17:37.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:37.698 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj572 testdata.548953 2024-02-05T06:17:37.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:37.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj573 testdata.548953 2024-02-05T06:17:37.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:37.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj574 testdata.548953 2024-02-05T06:17:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:38.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj575 testdata.548953 2024-02-05T06:17:38.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:38.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj576 testdata.548953 2024-02-05T06:17:38.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:38.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj577 testdata.548953 2024-02-05T06:17:38.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:38.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj578 testdata.548953 2024-02-05T06:17:38.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:38.695 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj579 testdata.548953 2024-02-05T06:17:38.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:38.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj580 testdata.548953 2024-02-05T06:17:38.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:38.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj581 testdata.548953 2024-02-05T06:17:39.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:39.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj582 testdata.548953 2024-02-05T06:17:39.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:39.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj583 testdata.548953 2024-02-05T06:17:39.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:39.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj584 testdata.548953 2024-02-05T06:17:39.508 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:39.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj585 testdata.548953 2024-02-05T06:17:39.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:39.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj586 testdata.548953 2024-02-05T06:17:39.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:39.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj587 testdata.548953 2024-02-05T06:17:40.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:40.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj588 testdata.548953 2024-02-05T06:17:40.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:40.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj589 testdata.548953 2024-02-05T06:17:40.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:40.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj590 testdata.548953 2024-02-05T06:17:40.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:40.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj591 testdata.548953 2024-02-05T06:17:40.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:40.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj592 testdata.548953 2024-02-05T06:17:40.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:40.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj593 testdata.548953 2024-02-05T06:17:40.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:40.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj594 testdata.548953 2024-02-05T06:17:41.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:41.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj595 testdata.548953 2024-02-05T06:17:41.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:41.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj596 testdata.548953 2024-02-05T06:17:41.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:41.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj597 testdata.548953 2024-02-05T06:17:41.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:41.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj598 testdata.548953 2024-02-05T06:17:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:41.595 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj599 testdata.548953 2024-02-05T06:17:41.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:41.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj600 testdata.548953 2024-02-05T06:17:41.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:41.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj601 testdata.548953 2024-02-05T06:17:42.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:42.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj602 testdata.548953 2024-02-05T06:17:42.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:42.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj603 testdata.548953 2024-02-05T06:17:42.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:42.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj604 testdata.548953 2024-02-05T06:17:42.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:42.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj605 testdata.548953 2024-02-05T06:17:42.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:42.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj606 testdata.548953 2024-02-05T06:17:42.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:42.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj607 testdata.548953 2024-02-05T06:17:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:42.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj608 testdata.548953 2024-02-05T06:17:43.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:43.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj609 testdata.548953 2024-02-05T06:17:43.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:43.163 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj610 testdata.548953 2024-02-05T06:17:43.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:43.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj611 testdata.548953 2024-02-05T06:17:43.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:43.431 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj612 testdata.548953 2024-02-05T06:17:43.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:43.574 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj613 testdata.548953 2024-02-05T06:17:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj614 testdata.548953 2024-02-05T06:17:43.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:43.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj615 testdata.548953 2024-02-05T06:17:44.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj616 testdata.548953 2024-02-05T06:17:44.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.162 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj617 testdata.548953 2024-02-05T06:17:44.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj618 testdata.548953 2024-02-05T06:17:44.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.429 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj619 testdata.548953 2024-02-05T06:17:44.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj620 testdata.548953 2024-02-05T06:17:44.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.714 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj621 testdata.548953 2024-02-05T06:17:44.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj622 testdata.548953 2024-02-05T06:17:44.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:44.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj623 testdata.548953 2024-02-05T06:17:45.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:45.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj624 testdata.548953 2024-02-05T06:17:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:45.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj625 testdata.548953 2024-02-05T06:17:45.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:45.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj626 testdata.548953 2024-02-05T06:17:45.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:45.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj627 testdata.548953 2024-02-05T06:17:45.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:45.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj628 testdata.548953 2024-02-05T06:17:45.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:45.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj629 testdata.548953 2024-02-05T06:17:46.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj630 testdata.548953 2024-02-05T06:17:46.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj631 testdata.548953 2024-02-05T06:17:46.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj632 testdata.548953 2024-02-05T06:17:46.449 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj633 testdata.548953 2024-02-05T06:17:46.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj634 testdata.548953 2024-02-05T06:17:46.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj635 testdata.548953 2024-02-05T06:17:46.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj636 testdata.548953 2024-02-05T06:17:46.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:46.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj637 testdata.548953 2024-02-05T06:17:47.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:47.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj638 testdata.548953 2024-02-05T06:17:47.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:47.254 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj639 testdata.548953 2024-02-05T06:17:47.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:47.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj640 testdata.548953 2024-02-05T06:17:47.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:47.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj641 testdata.548953 2024-02-05T06:17:47.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:47.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj642 testdata.548953 2024-02-05T06:17:47.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:47.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj643 testdata.548953 2024-02-05T06:17:48.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj644 testdata.548953 2024-02-05T06:17:48.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj645 testdata.548953 2024-02-05T06:17:48.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj646 testdata.548953 2024-02-05T06:17:48.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj647 testdata.548953 2024-02-05T06:17:48.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj648 testdata.548953 2024-02-05T06:17:48.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj649 testdata.548953 2024-02-05T06:17:48.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj650 testdata.548953 2024-02-05T06:17:48.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:48.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj651 testdata.548953 2024-02-05T06:17:49.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:49.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj652 testdata.548953 2024-02-05T06:17:49.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:49.283 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj653 testdata.548953 2024-02-05T06:17:49.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:49.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj654 testdata.548953 2024-02-05T06:17:49.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:49.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj655 testdata.548953 2024-02-05T06:17:49.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:49.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj656 testdata.548953 2024-02-05T06:17:49.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:49.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj657 testdata.548953 2024-02-05T06:17:50.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj658 testdata.548953 2024-02-05T06:17:50.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj659 testdata.548953 2024-02-05T06:17:50.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj660 testdata.548953 2024-02-05T06:17:50.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.448 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj661 testdata.548953 2024-02-05T06:17:50.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj662 testdata.548953 2024-02-05T06:17:50.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj663 testdata.548953 2024-02-05T06:17:50.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj664 testdata.548953 2024-02-05T06:17:50.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:50.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj665 testdata.548953 2024-02-05T06:17:51.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:51.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj666 testdata.548953 2024-02-05T06:17:51.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:51.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj667 testdata.548953 2024-02-05T06:17:51.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:51.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj668 testdata.548953 2024-02-05T06:17:51.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:51.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj669 testdata.548953 2024-02-05T06:17:51.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:51.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj670 testdata.548953 2024-02-05T06:17:51.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:51.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj671 testdata.548953 2024-02-05T06:17:51.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:51.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj672 testdata.548953 2024-02-05T06:17:52.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:52.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj673 testdata.548953 2024-02-05T06:17:52.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:52.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj674 testdata.548953 2024-02-05T06:17:52.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:52.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj675 testdata.548953 2024-02-05T06:17:52.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:52.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj676 testdata.548953 2024-02-05T06:17:52.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:52.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj677 testdata.548953 2024-02-05T06:17:52.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:52.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj678 testdata.548953 2024-02-05T06:17:52.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:52.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj679 testdata.548953 2024-02-05T06:17:53.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:53.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj680 testdata.548953 2024-02-05T06:17:53.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:53.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj681 testdata.548953 2024-02-05T06:17:53.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:53.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj682 testdata.548953 2024-02-05T06:17:53.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:53.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj683 testdata.548953 2024-02-05T06:17:53.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:53.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj684 testdata.548953 2024-02-05T06:17:53.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:53.786 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj685 testdata.548953 2024-02-05T06:17:53.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:53.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj686 testdata.548953 2024-02-05T06:17:54.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:54.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj687 testdata.548953 2024-02-05T06:17:54.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:54.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj688 testdata.548953 2024-02-05T06:17:54.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:54.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj689 testdata.548953 2024-02-05T06:17:54.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:54.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj690 testdata.548953 2024-02-05T06:17:54.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:54.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj691 testdata.548953 2024-02-05T06:17:54.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:54.766 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj692 testdata.548953 2024-02-05T06:17:54.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:54.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj693 testdata.548953 2024-02-05T06:17:55.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:55.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj694 testdata.548953 2024-02-05T06:17:55.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:55.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj695 testdata.548953 2024-02-05T06:17:55.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:55.319 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj696 testdata.548953 2024-02-05T06:17:55.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:55.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj697 testdata.548953 2024-02-05T06:17:55.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:55.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj698 testdata.548953 2024-02-05T06:17:55.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:55.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj699 testdata.548953 2024-02-05T06:17:55.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:55.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj700 testdata.548953 2024-02-05T06:17:56.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj701 testdata.548953 2024-02-05T06:17:56.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj702 testdata.548953 2024-02-05T06:17:56.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj703 testdata.548953 2024-02-05T06:17:56.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj704 testdata.548953 2024-02-05T06:17:56.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj705 testdata.548953 2024-02-05T06:17:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj706 testdata.548953 2024-02-05T06:17:56.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj707 testdata.548953 2024-02-05T06:17:56.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:56.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj708 testdata.548953 2024-02-05T06:17:57.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:57.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj709 testdata.548953 2024-02-05T06:17:57.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:57.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj710 testdata.548953 2024-02-05T06:17:57.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:57.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj711 testdata.548953 2024-02-05T06:17:57.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:57.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj712 testdata.548953 2024-02-05T06:17:57.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:57.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj713 testdata.548953 2024-02-05T06:17:57.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:57.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj714 testdata.548953 2024-02-05T06:17:58.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj715 testdata.548953 2024-02-05T06:17:58.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.153 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj716 testdata.548953 2024-02-05T06:17:58.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj717 testdata.548953 2024-02-05T06:17:58.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.438 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj718 testdata.548953 2024-02-05T06:17:58.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.572 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj719 testdata.548953 2024-02-05T06:17:58.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj720 testdata.548953 2024-02-05T06:17:58.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj721 testdata.548953 2024-02-05T06:17:58.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:58.974 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj722 testdata.548953 2024-02-05T06:17:59.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:59.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj723 testdata.548953 2024-02-05T06:17:59.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:59.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj724 testdata.548953 2024-02-05T06:17:59.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:59.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj725 testdata.548953 2024-02-05T06:17:59.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:17:59.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj726 testdata.548953 2024-02-05T06:18:00.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:00.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj727 testdata.548953 2024-02-05T06:18:00.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:00.233 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj728 testdata.548953 2024-02-05T06:18:00.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:00.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj729 testdata.548953 2024-02-05T06:18:00.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:00.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj730 testdata.548953 2024-02-05T06:18:00.687 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:00.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj731 testdata.548953 2024-02-05T06:18:00.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:00.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj732 testdata.548953 2024-02-05T06:18:00.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:00.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj733 testdata.548953 2024-02-05T06:18:01.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:01.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj734 testdata.548953 2024-02-05T06:18:01.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:01.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj735 testdata.548953 2024-02-05T06:18:01.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:01.381 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj736 testdata.548953 2024-02-05T06:18:01.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:01.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj737 testdata.548953 2024-02-05T06:18:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:01.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj738 testdata.548953 2024-02-05T06:18:01.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:01.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj739 testdata.548953 2024-02-05T06:18:01.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:01.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj740 testdata.548953 2024-02-05T06:18:02.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:02.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj741 testdata.548953 2024-02-05T06:18:02.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:02.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj742 testdata.548953 2024-02-05T06:18:02.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:02.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj743 testdata.548953 2024-02-05T06:18:02.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:02.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj744 testdata.548953 2024-02-05T06:18:02.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:02.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj745 testdata.548953 2024-02-05T06:18:02.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:02.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj746 testdata.548953 2024-02-05T06:18:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:02.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj747 testdata.548953 2024-02-05T06:18:03.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:03.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj748 testdata.548953 2024-02-05T06:18:03.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:03.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj749 testdata.548953 2024-02-05T06:18:03.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:03.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj750 testdata.548953 2024-02-05T06:18:03.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:03.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj751 testdata.548953 2024-02-05T06:18:03.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:03.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj752 testdata.548953 2024-02-05T06:18:03.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:03.763 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj753 testdata.548953 2024-02-05T06:18:03.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:03.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj754 testdata.548953 2024-02-05T06:18:04.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:04.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj755 testdata.548953 2024-02-05T06:18:04.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:04.173 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj756 testdata.548953 2024-02-05T06:18:04.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:04.316 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj757 testdata.548953 2024-02-05T06:18:04.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:04.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj758 testdata.548953 2024-02-05T06:18:04.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:04.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj759 testdata.548953 2024-02-05T06:18:04.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:04.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj760 testdata.548953 2024-02-05T06:18:04.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:04.860 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj761 testdata.548953 2024-02-05T06:18:05.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:05.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj762 testdata.548953 2024-02-05T06:18:05.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:05.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj763 testdata.548953 2024-02-05T06:18:05.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:05.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj764 testdata.548953 2024-02-05T06:18:05.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:05.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj765 testdata.548953 2024-02-05T06:18:05.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:05.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj766 testdata.548953 2024-02-05T06:18:05.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:05.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj767 testdata.548953 2024-02-05T06:18:05.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:05.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj768 testdata.548953 2024-02-05T06:18:06.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:06.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj769 testdata.548953 2024-02-05T06:18:06.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:06.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj770 testdata.548953 2024-02-05T06:18:06.427 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:06.428 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj771 testdata.548953 2024-02-05T06:18:06.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:06.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj772 testdata.548953 2024-02-05T06:18:06.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:06.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj773 testdata.548953 2024-02-05T06:18:06.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:06.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj774 testdata.548953 2024-02-05T06:18:06.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:06.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj775 testdata.548953 2024-02-05T06:18:07.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:07.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj776 testdata.548953 2024-02-05T06:18:07.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:07.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj777 testdata.548953 2024-02-05T06:18:07.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:07.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj778 testdata.548953 2024-02-05T06:18:07.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:07.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj779 testdata.548953 2024-02-05T06:18:07.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:07.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj780 testdata.548953 2024-02-05T06:18:07.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:07.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj781 testdata.548953 2024-02-05T06:18:07.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:07.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj782 testdata.548953 2024-02-05T06:18:08.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:08.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj783 testdata.548953 2024-02-05T06:18:08.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:08.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj784 testdata.548953 2024-02-05T06:18:08.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:08.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj785 testdata.548953 2024-02-05T06:18:08.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:08.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj786 testdata.548953 2024-02-05T06:18:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:08.699 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj787 testdata.548953 2024-02-05T06:18:08.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:08.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj788 testdata.548953 2024-02-05T06:18:08.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:08.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj789 testdata.548953 2024-02-05T06:18:09.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:09.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj790 testdata.548953 2024-02-05T06:18:09.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:09.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj791 testdata.548953 2024-02-05T06:18:09.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:09.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj792 testdata.548953 2024-02-05T06:18:09.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:09.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj793 testdata.548953 2024-02-05T06:18:09.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:09.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj794 testdata.548953 2024-02-05T06:18:09.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:09.848 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj795 testdata.548953 2024-02-05T06:18:09.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:09.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj796 testdata.548953 2024-02-05T06:18:10.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:10.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj797 testdata.548953 2024-02-05T06:18:10.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:10.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj798 testdata.548953 2024-02-05T06:18:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:10.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj799 testdata.548953 2024-02-05T06:18:10.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:10.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj800 testdata.548953 2024-02-05T06:18:10.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:10.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj801 testdata.548953 2024-02-05T06:18:10.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:10.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj802 testdata.548953 2024-02-05T06:18:10.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:10.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj803 testdata.548953 2024-02-05T06:18:11.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:11.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj804 testdata.548953 2024-02-05T06:18:11.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:11.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj805 testdata.548953 2024-02-05T06:18:11.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:11.382 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj806 testdata.548953 2024-02-05T06:18:11.516 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:11.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj807 testdata.548953 2024-02-05T06:18:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:11.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj808 testdata.548953 2024-02-05T06:18:11.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:11.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj809 testdata.548953 2024-02-05T06:18:12.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.011 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj810 testdata.548953 2024-02-05T06:18:12.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj811 testdata.548953 2024-02-05T06:18:12.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj812 testdata.548953 2024-02-05T06:18:12.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.422 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj813 testdata.548953 2024-02-05T06:18:12.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj814 testdata.548953 2024-02-05T06:18:12.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj815 testdata.548953 2024-02-05T06:18:12.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj816 testdata.548953 2024-02-05T06:18:12.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:12.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj817 testdata.548953 2024-02-05T06:18:13.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:13.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj818 testdata.548953 2024-02-05T06:18:13.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:13.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj819 testdata.548953 2024-02-05T06:18:13.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:13.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj820 testdata.548953 2024-02-05T06:18:13.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:13.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj821 testdata.548953 2024-02-05T06:18:13.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:13.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj822 testdata.548953 2024-02-05T06:18:13.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:13.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj823 testdata.548953 2024-02-05T06:18:13.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:13.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj824 testdata.548953 2024-02-05T06:18:14.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:14.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj825 testdata.548953 2024-02-05T06:18:14.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:14.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj826 testdata.548953 2024-02-05T06:18:14.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:14.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj827 testdata.548953 2024-02-05T06:18:14.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:14.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj828 testdata.548953 2024-02-05T06:18:14.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:14.677 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj829 testdata.548953 2024-02-05T06:18:14.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:14.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj830 testdata.548953 2024-02-05T06:18:14.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:14.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj831 testdata.548953 2024-02-05T06:18:15.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:15.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj832 testdata.548953 2024-02-05T06:18:15.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:15.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj833 testdata.548953 2024-02-05T06:18:15.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:15.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj834 testdata.548953 2024-02-05T06:18:15.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:15.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj835 testdata.548953 2024-02-05T06:18:15.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:15.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj836 testdata.548953 2024-02-05T06:18:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:15.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj837 testdata.548953 2024-02-05T06:18:16.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj838 testdata.548953 2024-02-05T06:18:16.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj839 testdata.548953 2024-02-05T06:18:16.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj840 testdata.548953 2024-02-05T06:18:16.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj841 testdata.548953 2024-02-05T06:18:16.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj842 testdata.548953 2024-02-05T06:18:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj843 testdata.548953 2024-02-05T06:18:16.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj844 testdata.548953 2024-02-05T06:18:16.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:16.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj845 testdata.548953 2024-02-05T06:18:17.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:17.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj846 testdata.548953 2024-02-05T06:18:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:17.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj847 testdata.548953 2024-02-05T06:18:17.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:17.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj848 testdata.548953 2024-02-05T06:18:17.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:17.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj849 testdata.548953 2024-02-05T06:18:17.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:17.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj850 testdata.548953 2024-02-05T06:18:17.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:17.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj851 testdata.548953 2024-02-05T06:18:18.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj852 testdata.548953 2024-02-05T06:18:18.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.139 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj853 testdata.548953 2024-02-05T06:18:18.281 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.282 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj854 testdata.548953 2024-02-05T06:18:18.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.416 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj855 testdata.548953 2024-02-05T06:18:18.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj856 testdata.548953 2024-02-05T06:18:18.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj857 testdata.548953 2024-02-05T06:18:18.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj858 testdata.548953 2024-02-05T06:18:18.960 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:18.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj859 testdata.548953 2024-02-05T06:18:19.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:19.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj860 testdata.548953 2024-02-05T06:18:19.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:19.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj861 testdata.548953 2024-02-05T06:18:19.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:19.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj862 testdata.548953 2024-02-05T06:18:19.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:19.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj863 testdata.548953 2024-02-05T06:18:19.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:19.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj864 testdata.548953 2024-02-05T06:18:19.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:19.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj865 testdata.548953 2024-02-05T06:18:19.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:19.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj866 testdata.548953 2024-02-05T06:18:20.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:20.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj867 testdata.548953 2024-02-05T06:18:20.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:20.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj868 testdata.548953 2024-02-05T06:18:20.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:20.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj869 testdata.548953 2024-02-05T06:18:20.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:20.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj870 testdata.548953 2024-02-05T06:18:20.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:20.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj871 testdata.548953 2024-02-05T06:18:20.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:20.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj872 testdata.548953 2024-02-05T06:18:20.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:20.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj873 testdata.548953 2024-02-05T06:18:21.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:21.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj874 testdata.548953 2024-02-05T06:18:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:21.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj875 testdata.548953 2024-02-05T06:18:21.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:21.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj876 testdata.548953 2024-02-05T06:18:21.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:21.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj877 testdata.548953 2024-02-05T06:18:21.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:21.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj878 testdata.548953 2024-02-05T06:18:21.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:21.870 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj879 testdata.548953 2024-02-05T06:18:22.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.012 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj880 testdata.548953 2024-02-05T06:18:22.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.146 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj881 testdata.548953 2024-02-05T06:18:22.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj882 testdata.548953 2024-02-05T06:18:22.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj883 testdata.548953 2024-02-05T06:18:22.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.557 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj884 testdata.548953 2024-02-05T06:18:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj885 testdata.548953 2024-02-05T06:18:22.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.825 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj886 testdata.548953 2024-02-05T06:18:22.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:22.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj887 testdata.548953 2024-02-05T06:18:23.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:23.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj888 testdata.548953 2024-02-05T06:18:23.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:23.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj889 testdata.548953 2024-02-05T06:18:23.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:23.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj890 testdata.548953 2024-02-05T06:18:23.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:23.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj891 testdata.548953 2024-02-05T06:18:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj892 testdata.548953 2024-02-05T06:18:23.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:23.857 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj893 testdata.548953 2024-02-05T06:18:23.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:23.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj894 testdata.548953 2024-02-05T06:18:24.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:24.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj895 testdata.548953 2024-02-05T06:18:24.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:24.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj896 testdata.548953 2024-02-05T06:18:24.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:24.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj897 testdata.548953 2024-02-05T06:18:24.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:24.536 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj898 testdata.548953 2024-02-05T06:18:24.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:24.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj899 testdata.548953 2024-02-05T06:18:24.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:24.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj900 testdata.548953 2024-02-05T06:18:24.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:24.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj901 testdata.548953 2024-02-05T06:18:25.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:25.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj902 testdata.548953 2024-02-05T06:18:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:25.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj903 testdata.548953 2024-02-05T06:18:25.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:25.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj904 testdata.548953 2024-02-05T06:18:25.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:25.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj905 testdata.548953 2024-02-05T06:18:25.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:25.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj906 testdata.548953 2024-02-05T06:18:25.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:25.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj907 testdata.548953 2024-02-05T06:18:25.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:25.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj908 testdata.548953 2024-02-05T06:18:26.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:26.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj909 testdata.548953 2024-02-05T06:18:26.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:26.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj910 testdata.548953 2024-02-05T06:18:26.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:26.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj911 testdata.548953 2024-02-05T06:18:26.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:26.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj912 testdata.548953 2024-02-05T06:18:26.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:26.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj913 testdata.548953 2024-02-05T06:18:26.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:26.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj914 testdata.548953 2024-02-05T06:18:26.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:26.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj915 testdata.548953 2024-02-05T06:18:27.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:27.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj916 testdata.548953 2024-02-05T06:18:27.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:27.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj917 testdata.548953 2024-02-05T06:18:27.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:27.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj918 testdata.548953 2024-02-05T06:18:27.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:27.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj919 testdata.548953 2024-02-05T06:18:27.582 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:27.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj920 testdata.548953 2024-02-05T06:18:27.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:27.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj921 testdata.548953 2024-02-05T06:18:27.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:27.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj922 testdata.548953 2024-02-05T06:18:28.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:28.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj923 testdata.548953 2024-02-05T06:18:28.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:28.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj924 testdata.548953 2024-02-05T06:18:28.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:28.338 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj925 testdata.548953 2024-02-05T06:18:28.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:28.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj926 testdata.548953 2024-02-05T06:18:28.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:28.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj927 testdata.548953 2024-02-05T06:18:28.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:28.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj928 testdata.548953 2024-02-05T06:18:28.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:28.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj929 testdata.548953 2024-02-05T06:18:29.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:29.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj930 testdata.548953 2024-02-05T06:18:29.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:29.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj931 testdata.548953 2024-02-05T06:18:29.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:29.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj932 testdata.548953 2024-02-05T06:18:29.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:29.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj933 testdata.548953 2024-02-05T06:18:29.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:29.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj934 testdata.548953 2024-02-05T06:18:29.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:29.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj935 testdata.548953 2024-02-05T06:18:29.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:29.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj936 testdata.548953 2024-02-05T06:18:30.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:30.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj937 testdata.548953 2024-02-05T06:18:30.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:30.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj938 testdata.548953 2024-02-05T06:18:30.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:30.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj939 testdata.548953 2024-02-05T06:18:30.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:30.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj940 testdata.548953 2024-02-05T06:18:30.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:30.559 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj941 testdata.548953 2024-02-05T06:18:30.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:30.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj942 testdata.548953 2024-02-05T06:18:31.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:31.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj943 testdata.548953 2024-02-05T06:18:31.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:31.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj944 testdata.548953 2024-02-05T06:18:31.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:31.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj945 testdata.548953 2024-02-05T06:18:31.621 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:31.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj946 testdata.548953 2024-02-05T06:18:31.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:31.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj947 testdata.548953 2024-02-05T06:18:31.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:31.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj948 testdata.548953 2024-02-05T06:18:32.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:32.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj949 testdata.548953 2024-02-05T06:18:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj950 testdata.548953 2024-02-05T06:18:32.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:32.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj951 testdata.548953 2024-02-05T06:18:32.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:32.502 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj952 testdata.548953 2024-02-05T06:18:32.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:32.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj953 testdata.548953 2024-02-05T06:18:32.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:32.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj954 testdata.548953 2024-02-05T06:18:32.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:32.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj955 testdata.548953 2024-02-05T06:18:33.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:33.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj956 testdata.548953 2024-02-05T06:18:33.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:33.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj957 testdata.548953 2024-02-05T06:18:33.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:33.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj958 testdata.548953 2024-02-05T06:18:33.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:33.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj959 testdata.548953 2024-02-05T06:18:33.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:33.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj960 testdata.548953 2024-02-05T06:18:33.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:33.826 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj961 testdata.548953 2024-02-05T06:18:33.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:33.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj962 testdata.548953 2024-02-05T06:18:34.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:34.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj963 testdata.548953 2024-02-05T06:18:34.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:34.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj964 testdata.548953 2024-02-05T06:18:34.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:34.437 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj965 testdata.548953 2024-02-05T06:18:34.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:34.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj966 testdata.548953 2024-02-05T06:18:34.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:34.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj967 testdata.548953 2024-02-05T06:18:34.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:34.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj968 testdata.548953 2024-02-05T06:18:35.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.015 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj969 testdata.548953 2024-02-05T06:18:35.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj970 testdata.548953 2024-02-05T06:18:35.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj971 testdata.548953 2024-02-05T06:18:35.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj972 testdata.548953 2024-02-05T06:18:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj973 testdata.548953 2024-02-05T06:18:35.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj974 testdata.548953 2024-02-05T06:18:35.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj975 testdata.548953 2024-02-05T06:18:35.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:35.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj976 testdata.548953 2024-02-05T06:18:36.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:36.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj977 testdata.548953 2024-02-05T06:18:36.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:36.273 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj978 testdata.548953 2024-02-05T06:18:36.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:36.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj979 testdata.548953 2024-02-05T06:18:36.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:36.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj980 testdata.548953 2024-02-05T06:18:36.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:36.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj981 testdata.548953 2024-02-05T06:18:36.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:36.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj982 testdata.548953 2024-02-05T06:18:36.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:36.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj983 testdata.548953 2024-02-05T06:18:37.160 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:37.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj984 testdata.548953 2024-02-05T06:18:37.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:37.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj985 testdata.548953 2024-02-05T06:18:37.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:37.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj986 testdata.548953 2024-02-05T06:18:37.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:37.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj987 testdata.548953 2024-02-05T06:18:37.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:37.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj988 testdata.548953 2024-02-05T06:18:37.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:37.891 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj989 testdata.548953 2024-02-05T06:18:38.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:38.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj990 testdata.548953 2024-02-05T06:18:38.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:38.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj991 testdata.548953 2024-02-05T06:18:38.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:38.301 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj992 testdata.548953 2024-02-05T06:18:38.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:38.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj993 testdata.548953 2024-02-05T06:18:38.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:38.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj994 testdata.548953 2024-02-05T06:18:38.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:38.753 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj995 testdata.548953 2024-02-05T06:18:38.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:38.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj996 testdata.548953 2024-02-05T06:18:39.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:39.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj997 testdata.548953 2024-02-05T06:18:39.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:39.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj998 testdata.548953 2024-02-05T06:18:39.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:39.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj999 testdata.548953 2024-02-05T06:18:39.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:335: _scrub_abort: for i in `seq 1 $objects` 2024-02-05T06:18:39.426 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: rados -p test put obj1000 testdata.548953 2024-02-05T06:18:39.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: rm -f testdata.548953 2024-02-05T06:18:39.561 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: get_primary test obj1 2024-02-05T06:18:39.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:18:39.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:18:39.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:18:39.562 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:18:39.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:341: _scrub_abort: local primary=1 2024-02-05T06:18:39.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: local pgid=1.0 2024-02-05T06:18:39.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2024-02-05T06:18:39.936 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:18:39.936 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:18:39.936 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:18:39.936 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-22T06:16:59.961518+0000" 2024-02-05T06:18:39.937 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:18:39.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: set -o pipefail 2024-02-05T06:18:39.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:348: _scrub_abort: found=no 2024-02-05T06:18:39.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: seq 0 200 2024-02-05T06:18:39.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:18:39.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: flush_pg_stats 2024-02-05T06:18:39.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:39.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:40.207 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:40.207 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:18:40.207 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:18:40.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:40.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:40.208 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:40.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836517 2024-02-05T06:18:40.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836517 2024-02-05T06:18:40.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517' 2024-02-05T06:18:40.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:40.308 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:40.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672995 2024-02-05T06:18:40.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672995 2024-02-05T06:18:40.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-42949672995' 2024-02-05T06:18:40.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:40.409 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:40.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509473 2024-02-05T06:18:40.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509473 2024-02-05T06:18:40.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-42949672995 2-64424509473' 2024-02-05T06:18:40.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:40.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836517 2024-02-05T06:18:40.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:40.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:40.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836517 2024-02-05T06:18:40.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:40.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836517 2024-02-05T06:18:40.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-02-05T06:18:40.513 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836517 2024-02-05T06:18:40.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:40.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836517 2024-02-05T06:18:40.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:41.776 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:18:41.777 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836517 2024-02-05T06:18:42.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:42.044 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672995 2024-02-05T06:18:42.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:42.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:42.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672995 2024-02-05T06:18:42.046 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:42.047 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672995 2024-02-05T06:18:42.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672995 2024-02-05T06:18:42.047 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-02-05T06:18:42.047 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:42.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672995 -lt 42949672995 2024-02-05T06:18:42.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:42.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509473 2024-02-05T06:18:42.311 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:42.312 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:42.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509473 2024-02-05T06:18:42.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:42.314 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509473 2024-02-05T06:18:42.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509473 2024-02-05T06:18:42.314 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2024-02-05T06:18:42.314 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:42.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509473 -lt 64424509473 2024-02-05T06:18:42.579 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: ceph pg dump pgs 2024-02-05T06:18:42.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep '^1.0' 2024-02-05T06:18:42.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep -q scrubbing 2024-02-05T06:18:42.828 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:18:42.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:18:42.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: flush_pg_stats 2024-02-05T06:18:42.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:42.840 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:43.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:43.102 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:18:43.102 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:18:43.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:43.102 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:43.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:43.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-02-05T06:18:43.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-02-05T06:18:43.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-02-05T06:18:43.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:43.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:43.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672996 2024-02-05T06:18:43.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672996 2024-02-05T06:18:43.305 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672996' 2024-02-05T06:18:43.306 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:43.306 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:43.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509474 2024-02-05T06:18:43.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509474 2024-02-05T06:18:43.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-42949672996 2-64424509474' 2024-02-05T06:18:43.406 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:43.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-02-05T06:18:43.407 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:43.407 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:43.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-02-05T06:18:43.408 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:43.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-02-05T06:18:43.409 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836519 2024-02-05T06:18:43.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-02-05T06:18:43.410 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:43.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836519 2024-02-05T06:18:43.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:43.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672996 2024-02-05T06:18:43.705 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:43.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:43.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672996 2024-02-05T06:18:43.707 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672996 2024-02-05T06:18:43.708 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672996 2024-02-05T06:18:43.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-02-05T06:18:43.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672996 -lt 42949672996 2024-02-05T06:18:43.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509474 2024-02-05T06:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:43.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:43.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509474 2024-02-05T06:18:43.974 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509474 2024-02-05T06:18:43.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-02-05T06:18:43.976 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509474 2024-02-05T06:18:43.976 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:44.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509474 -lt 64424509474 2024-02-05T06:18:44.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: ceph pg dump pgs 2024-02-05T06:18:44.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep '^1.0' 2024-02-05T06:18:44.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:352: _scrub_abort: grep -q scrubbing 2024-02-05T06:18:44.495 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:18:44.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: found=yes 2024-02-05T06:18:44.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: break 2024-02-05T06:18:44.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:359: _scrub_abort: set +o pipefail 2024-02-05T06:18:44.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:361: _scrub_abort: test yes = no 2024-02-05T06:18:44.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:367: _scrub_abort: ceph osd set nodeep-scrub 2024-02-05T06:18:44.879 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:18:44.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:368: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-02-05T06:18:44.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: ceph osd set noscrub 2024-02-05T06:18:45.389 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:18:45.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:374: _scrub_abort: set -o pipefail 2024-02-05T06:18:45.401 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: seq 0 200 2024-02-05T06:18:45.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:18:45.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:18:45.402 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:45.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:45.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:45.696 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:18:45.696 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:18:45.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:45.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:45.696 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:45.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836520 2024-02-05T06:18:45.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836520 2024-02-05T06:18:45.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520' 2024-02-05T06:18:45.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:45.797 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:45.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672998 2024-02-05T06:18:45.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672998 2024-02-05T06:18:45.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672998' 2024-02-05T06:18:45.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:45.897 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:45.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509476 2024-02-05T06:18:45.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509476 2024-02-05T06:18:45.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836520 1-42949672998 2-64424509476' 2024-02-05T06:18:45.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:45.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836520 2024-02-05T06:18:45.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:45.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:45.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836520 2024-02-05T06:18:45.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:45.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836520 2024-02-05T06:18:45.999 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836520 2024-02-05T06:18:45.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-02-05T06:18:46.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:46.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-02-05T06:18:46.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:47.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:18:47.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:47.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836520 2024-02-05T06:18:47.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:48.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:18:48.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:48.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836520 2024-02-05T06:18:48.833 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:48.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672998 2024-02-05T06:18:48.834 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:48.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:48.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672998 2024-02-05T06:18:48.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:48.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672998 2024-02-05T06:18:48.836 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672998 2024-02-05T06:18:48.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-02-05T06:18:48.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:49.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672998 -lt 42949672998 2024-02-05T06:18:49.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:49.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509476 2024-02-05T06:18:49.100 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:49.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:49.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509476 2024-02-05T06:18:49.102 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:49.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509476 2024-02-05T06:18:49.103 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-02-05T06:18:49.103 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509476 2024-02-05T06:18:49.103 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:49.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509476 -lt 64424509476 2024-02-05T06:18:49.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:18:49.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:18:49.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:18:49.620 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:18:49.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:18:49.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:18:49.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:18:49.632 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:49.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:18:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:18:49.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:49.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:49.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:49.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836522 2024-02-05T06:18:49.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836522 2024-02-05T06:18:50.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522' 2024-02-05T06:18:50.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:50.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:50.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673000 2024-02-05T06:18:50.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673000 2024-02-05T06:18:50.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949673000' 2024-02-05T06:18:50.101 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:50.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:50.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509477 2024-02-05T06:18:50.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509477 2024-02-05T06:18:50.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836522 1-42949673000 2-64424509477' 2024-02-05T06:18:50.201 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:50.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836522 2024-02-05T06:18:50.201 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:50.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:50.202 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836522 2024-02-05T06:18:50.203 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:50.203 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836522 2024-02-05T06:18:50.204 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836522 2024-02-05T06:18:50.204 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2024-02-05T06:18:50.204 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:50.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836522 2024-02-05T06:18:50.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:51.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:18:51.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:51.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836522 2024-02-05T06:18:51.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:51.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673000 2024-02-05T06:18:51.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:51.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:51.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673000 2024-02-05T06:18:51.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:51.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673000 2024-02-05T06:18:51.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-02-05T06:18:51.745 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673000 2024-02-05T06:18:51.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:52.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673000 -lt 42949673000 2024-02-05T06:18:52.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:52.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509477 2024-02-05T06:18:52.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:52.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:52.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509477 2024-02-05T06:18:52.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:52.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509477 2024-02-05T06:18:52.035 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509477 2024-02-05T06:18:52.035 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-02-05T06:18:52.035 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:52.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509478 -lt 64424509477 2024-02-05T06:18:52.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:18:52.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:18:52.302 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:18:52.554 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:18:52.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:18:52.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:18:52.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:18:52.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:52.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:18:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:18:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:52.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:52.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:52.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-02-05T06:18:52.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-02-05T06:18:52.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-02-05T06:18:52.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:52.928 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:53.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673001 2024-02-05T06:18:53.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673001 2024-02-05T06:18:53.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-42949673001' 2024-02-05T06:18:53.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:53.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:53.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509479 2024-02-05T06:18:53.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509479 2024-02-05T06:18:53.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-42949673001 2-64424509479' 2024-02-05T06:18:53.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:53.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-02-05T06:18:53.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:53.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:53.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-02-05T06:18:53.130 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:53.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-02-05T06:18:53.132 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836523 2024-02-05T06:18:53.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-02-05T06:18:53.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:53.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836522 -lt 21474836523 2024-02-05T06:18:53.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:54.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:18:54.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:54.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836523 2024-02-05T06:18:54.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:54.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673001 2024-02-05T06:18:54.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:54.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:54.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673001 2024-02-05T06:18:54.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:54.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673001 2024-02-05T06:18:54.683 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673001 2024-02-05T06:18:54.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-02-05T06:18:54.683 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:54.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673001 -lt 42949673001 2024-02-05T06:18:54.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:54.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509479 2024-02-05T06:18:54.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:54.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:54.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509479 2024-02-05T06:18:54.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:54.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509479 2024-02-05T06:18:54.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-02-05T06:18:54.950 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509479 2024-02-05T06:18:54.950 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:55.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509479 -lt 64424509479 2024-02-05T06:18:55.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:18:55.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:18:55.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:18:55.464 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:18:55.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:18:55.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:18:55.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:18:55.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:55.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:18:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:18:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:55.739 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:55.739 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:55.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-02-05T06:18:55.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-02-05T06:18:55.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-02-05T06:18:55.869 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:55.870 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:55.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673003 2024-02-05T06:18:55.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673003 2024-02-05T06:18:55.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949673003' 2024-02-05T06:18:55.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:55.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:56.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509481 2024-02-05T06:18:56.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509481 2024-02-05T06:18:56.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-42949673003 2-64424509481' 2024-02-05T06:18:56.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:56.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-02-05T06:18:56.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:56.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:56.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-02-05T06:18:56.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:56.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-02-05T06:18:56.073 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836525 2024-02-05T06:18:56.073 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-02-05T06:18:56.073 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:56.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836524 -lt 21474836525 2024-02-05T06:18:56.336 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:18:57.337 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:18:57.337 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:57.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-02-05T06:18:57.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:57.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673003 2024-02-05T06:18:57.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:57.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:18:57.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673003 2024-02-05T06:18:57.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:57.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673003 2024-02-05T06:18:57.612 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673003 2024-02-05T06:18:57.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2024-02-05T06:18:57.612 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:18:57.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673003 -lt 42949673003 2024-02-05T06:18:57.881 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:57.881 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509481 2024-02-05T06:18:57.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:57.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:18:57.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509481 2024-02-05T06:18:57.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:57.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509481 2024-02-05T06:18:57.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-02-05T06:18:57.884 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509481 2024-02-05T06:18:57.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:18:58.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509481 -lt 64424509481 2024-02-05T06:18:58.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:18:58.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:18:58.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:18:58.401 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:18:58.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:18:58.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:18:58.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:18:58.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:18:58.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:18:58.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:18:58.691 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:18:58.691 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:18:58.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:18:58.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:58.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:18:58.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-02-05T06:18:58.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-02-05T06:18:58.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-02-05T06:18:58.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:58.805 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:18:58.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673004 2024-02-05T06:18:58.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673004 2024-02-05T06:18:58.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949673004' 2024-02-05T06:18:58.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:18:58.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:18:59.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509482 2024-02-05T06:18:59.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509482 2024-02-05T06:18:59.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-42949673004 2-64424509482' 2024-02-05T06:18:59.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:18:59.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-02-05T06:18:59.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:18:59.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:18:59.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-02-05T06:18:59.009 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:18:59.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-02-05T06:18:59.010 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836527 2024-02-05T06:18:59.010 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-02-05T06:18:59.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:18:59.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836527 2024-02-05T06:18:59.275 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:00.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:00.277 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:00.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836527 2024-02-05T06:19:00.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:00.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673004 2024-02-05T06:19:00.552 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:00.553 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:00.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673004 2024-02-05T06:19:00.554 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:00.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673004 2024-02-05T06:19:00.555 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673004 2024-02-05T06:19:00.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-02-05T06:19:00.555 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:00.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673005 -lt 42949673004 2024-02-05T06:19:00.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:00.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509482 2024-02-05T06:19:00.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:00.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:00.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509482 2024-02-05T06:19:00.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:00.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509482 2024-02-05T06:19:00.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-02-05T06:19:00.822 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509482 2024-02-05T06:19:00.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:01.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509482 -lt 64424509482 2024-02-05T06:19:01.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:01.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:01.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:01.336 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:01.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:01.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:01.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:01.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:01.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:01.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:01.613 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:01.613 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:01.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:01.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:01.613 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:01.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836528 2024-02-05T06:19:01.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836528 2024-02-05T06:19:01.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528' 2024-02-05T06:19:01.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:01.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:01.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673006 2024-02-05T06:19:01.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673006 2024-02-05T06:19:01.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-42949673006' 2024-02-05T06:19:01.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:01.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:01.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509484 2024-02-05T06:19:01.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509484 2024-02-05T06:19:01.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836528 1-42949673006 2-64424509484' 2024-02-05T06:19:01.930 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:01.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836528 2024-02-05T06:19:01.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:01.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:01.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836528 2024-02-05T06:19:01.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:01.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836528 2024-02-05T06:19:01.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-02-05T06:19:01.933 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836528 2024-02-05T06:19:01.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:02.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836528 2024-02-05T06:19:02.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:03.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:03.196 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:03.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836527 -lt 21474836528 2024-02-05T06:19:03.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:04.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:19:04.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:04.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836528 2024-02-05T06:19:04.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:04.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673006 2024-02-05T06:19:04.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:04.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:04.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673006 2024-02-05T06:19:04.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:04.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673006 2024-02-05T06:19:04.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2024-02-05T06:19:04.744 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673006 2024-02-05T06:19:04.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:05.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673006 -lt 42949673006 2024-02-05T06:19:05.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:05.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509484 2024-02-05T06:19:05.020 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:05.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:05.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509484 2024-02-05T06:19:05.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:05.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509484 2024-02-05T06:19:05.022 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509484 2024-02-05T06:19:05.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-02-05T06:19:05.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:05.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509484 -lt 64424509484 2024-02-05T06:19:05.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:05.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:05.286 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:05.535 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:05.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:05.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:05.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:05.546 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:05.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:05.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:05.818 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:05.818 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:05.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:05.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:05.819 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:05.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-02-05T06:19:05.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-02-05T06:19:05.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-02-05T06:19:05.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:05.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:06.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673008 2024-02-05T06:19:06.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673008 2024-02-05T06:19:06.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-42949673008' 2024-02-05T06:19:06.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:06.018 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:06.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509486 2024-02-05T06:19:06.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509486 2024-02-05T06:19:06.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509486' 2024-02-05T06:19:06.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:06.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-02-05T06:19:06.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:06.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:06.119 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-02-05T06:19:06.119 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:06.120 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836530 2024-02-05T06:19:06.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-02-05T06:19:06.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-02-05T06:19:06.120 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:06.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836529 -lt 21474836530 2024-02-05T06:19:06.385 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:07.386 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:07.386 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:07.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836530 2024-02-05T06:19:07.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:07.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673008 2024-02-05T06:19:07.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:07.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:07.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673008 2024-02-05T06:19:07.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:07.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673008 2024-02-05T06:19:07.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-02-05T06:19:07.663 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673008 2024-02-05T06:19:07.663 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:07.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673008 -lt 42949673008 2024-02-05T06:19:07.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:07.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509486 2024-02-05T06:19:07.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:07.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:07.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509486 2024-02-05T06:19:07.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:07.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509486 2024-02-05T06:19:07.931 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509486 2024-02-05T06:19:07.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2024-02-05T06:19:07.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:08.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509486 -lt 64424509486 2024-02-05T06:19:08.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:08.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:08.196 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:08.445 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:08.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:08.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:08.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:08.722 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:08.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-02-05T06:19:08.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-02-05T06:19:08.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-02-05T06:19:08.855 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:08.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:08.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673009 2024-02-05T06:19:08.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673009 2024-02-05T06:19:08.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-42949673009' 2024-02-05T06:19:08.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:08.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509487 2024-02-05T06:19:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509487 2024-02-05T06:19:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-42949673009 2-64424509487' 2024-02-05T06:19:09.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:09.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-02-05T06:19:09.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:09.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:09.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-02-05T06:19:09.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:09.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-02-05T06:19:09.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-02-05T06:19:09.059 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836532 2024-02-05T06:19:09.059 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:09.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836532 2024-02-05T06:19:09.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:10.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:10.323 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836532 2024-02-05T06:19:10.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:10.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673009 2024-02-05T06:19:10.591 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:10.591 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:10.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673009 2024-02-05T06:19:10.592 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:10.593 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673009 2024-02-05T06:19:10.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673009 2024-02-05T06:19:10.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-02-05T06:19:10.594 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:11.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673010 -lt 42949673009 2024-02-05T06:19:11.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:11.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509487 2024-02-05T06:19:11.701 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:11.701 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:11.702 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509487 2024-02-05T06:19:11.702 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:11.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509487 2024-02-05T06:19:11.703 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509487 2024-02-05T06:19:11.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2024-02-05T06:19:11.704 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:11.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509488 -lt 64424509487 2024-02-05T06:19:11.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:11.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:11.977 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:12.226 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:12.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:12.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:12.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:12.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:12.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:12.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:12.500 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:12.501 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:12.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:12.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:12.501 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:12.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836533 2024-02-05T06:19:12.599 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836533 2024-02-05T06:19:12.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533' 2024-02-05T06:19:12.600 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:12.600 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:12.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673011 2024-02-05T06:19:12.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673011 2024-02-05T06:19:12.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-42949673011' 2024-02-05T06:19:12.700 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:12.700 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:12.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509489 2024-02-05T06:19:12.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509489 2024-02-05T06:19:12.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-42949673011 2-64424509489' 2024-02-05T06:19:12.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:12.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836533 2024-02-05T06:19:12.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:12.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:12.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836533 2024-02-05T06:19:12.829 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:12.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836533 2024-02-05T06:19:12.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-02-05T06:19:12.831 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836533 2024-02-05T06:19:12.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:13.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836532 -lt 21474836533 2024-02-05T06:19:13.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:14.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:14.096 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:14.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836533 2024-02-05T06:19:14.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:14.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673011 2024-02-05T06:19:14.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:14.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:14.373 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673011 2024-02-05T06:19:14.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:14.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673011 2024-02-05T06:19:14.375 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673011 2024-02-05T06:19:14.375 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-02-05T06:19:14.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:14.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673011 -lt 42949673011 2024-02-05T06:19:14.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:14.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509489 2024-02-05T06:19:14.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:14.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:14.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509489 2024-02-05T06:19:14.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:14.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509489 2024-02-05T06:19:14.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509489' 2024-02-05T06:19:14.646 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509489 2024-02-05T06:19:14.646 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:14.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509489 -lt 64424509489 2024-02-05T06:19:14.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:14.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:14.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:15.159 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:15.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:15.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:15.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:15.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:15.171 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:15.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:15.432 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:15.432 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:15.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:15.432 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:15.432 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:15.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-02-05T06:19:15.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-02-05T06:19:15.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-02-05T06:19:15.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:15.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:15.633 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673013 2024-02-05T06:19:15.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673013 2024-02-05T06:19:15.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-42949673013' 2024-02-05T06:19:15.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:15.634 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:15.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509491 2024-02-05T06:19:15.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509491 2024-02-05T06:19:15.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-42949673013 2-64424509491' 2024-02-05T06:19:15.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:15.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-02-05T06:19:15.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:15.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:15.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-02-05T06:19:15.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:15.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-02-05T06:19:15.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-02-05T06:19:15.744 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836535 2024-02-05T06:19:15.744 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836535 2024-02-05T06:19:16.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:17.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:17.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:17.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836535 2024-02-05T06:19:17.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:18.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:19:18.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:18.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836535 2024-02-05T06:19:18.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:18.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673013 2024-02-05T06:19:18.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:18.551 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:18.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673013 2024-02-05T06:19:18.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:18.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673013 2024-02-05T06:19:18.552 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2024-02-05T06:19:18.553 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673013 2024-02-05T06:19:18.553 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:18.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673013 -lt 42949673013 2024-02-05T06:19:18.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:18.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509491 2024-02-05T06:19:18.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:18.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:18.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509491 2024-02-05T06:19:18.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:18.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509491 2024-02-05T06:19:18.821 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509491 2024-02-05T06:19:18.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-02-05T06:19:18.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:19.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509491 -lt 64424509491 2024-02-05T06:19:19.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:19.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:19.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:19.339 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:19.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:19.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:19.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:19.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:19.351 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:19.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:19.636 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:19.637 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:19.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:19.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:19.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:19.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-02-05T06:19:19.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-02-05T06:19:19.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-02-05T06:19:19.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:19.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673015 2024-02-05T06:19:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673015 2024-02-05T06:19:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-42949673015' 2024-02-05T06:19:19.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:19.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:19.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509492 2024-02-05T06:19:19.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509492 2024-02-05T06:19:19.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-42949673015 2-64424509492' 2024-02-05T06:19:19.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:19.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-02-05T06:19:19.980 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:19.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:19.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-02-05T06:19:19.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:19.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-02-05T06:19:19.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-02-05T06:19:19.983 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836537 2024-02-05T06:19:19.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:20.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-02-05T06:19:20.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:21.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:21.247 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:21.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-02-05T06:19:21.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:22.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:19:22.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:22.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836537 -lt 21474836537 2024-02-05T06:19:22.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:22.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673015 2024-02-05T06:19:22.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:22.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:22.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673015 2024-02-05T06:19:22.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:22.801 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673015 2024-02-05T06:19:22.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673015 2024-02-05T06:19:22.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2024-02-05T06:19:22.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:23.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673015 -lt 42949673015 2024-02-05T06:19:23.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:23.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509492 2024-02-05T06:19:23.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:23.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:23.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509492 2024-02-05T06:19:23.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:23.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509492 2024-02-05T06:19:23.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509492' 2024-02-05T06:19:23.070 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509492 2024-02-05T06:19:23.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:23.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509493 -lt 64424509492 2024-02-05T06:19:23.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:23.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:23.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:23.615 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:23.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:23.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:23.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:23.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:23.626 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:23.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:23.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:24.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-02-05T06:19:24.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-02-05T06:19:24.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-02-05T06:19:24.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:24.008 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:24.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673016 2024-02-05T06:19:24.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673016 2024-02-05T06:19:24.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-42949673016' 2024-02-05T06:19:24.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:24.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509494 2024-02-05T06:19:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509494 2024-02-05T06:19:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-42949673016 2-64424509494' 2024-02-05T06:19:24.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:24.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-02-05T06:19:24.209 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:24.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:24.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-02-05T06:19:24.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:24.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-02-05T06:19:24.211 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836539 2024-02-05T06:19:24.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-02-05T06:19:24.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:24.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-02-05T06:19:24.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:25.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:25.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:25.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836539 2024-02-05T06:19:25.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:25.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673016 2024-02-05T06:19:25.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:25.760 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:25.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673016 2024-02-05T06:19:25.760 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:25.761 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673016 2024-02-05T06:19:25.761 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673016 2024-02-05T06:19:25.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2024-02-05T06:19:25.762 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:26.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673017 -lt 42949673016 2024-02-05T06:19:26.050 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:26.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509494 2024-02-05T06:19:26.050 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:26.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:26.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509494 2024-02-05T06:19:26.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:26.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509494 2024-02-05T06:19:26.053 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509494 2024-02-05T06:19:26.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-02-05T06:19:26.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:26.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509495 -lt 64424509494 2024-02-05T06:19:26.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:26.320 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:26.321 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:26.570 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:26.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:26.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:26.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:26.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:26.582 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:26.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:26.845 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:26.845 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:26.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:26.845 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:26.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:26.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836540 2024-02-05T06:19:26.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836540 2024-02-05T06:19:26.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540' 2024-02-05T06:19:26.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:26.945 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:27.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673018 2024-02-05T06:19:27.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673018 2024-02-05T06:19:27.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-42949673018' 2024-02-05T06:19:27.045 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:27.045 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:27.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509496 2024-02-05T06:19:27.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509496 2024-02-05T06:19:27.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836540 1-42949673018 2-64424509496' 2024-02-05T06:19:27.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:27.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836540 2024-02-05T06:19:27.145 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:27.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:27.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836540 2024-02-05T06:19:27.146 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:27.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836540 2024-02-05T06:19:27.147 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836540 2024-02-05T06:19:27.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-02-05T06:19:27.148 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:27.413 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836540 2024-02-05T06:19:27.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:28.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:28.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:28.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836540 -lt 21474836540 2024-02-05T06:19:28.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:28.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673018 2024-02-05T06:19:28.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:28.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:28.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673018 2024-02-05T06:19:28.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:28.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673018 2024-02-05T06:19:28.692 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673018 2024-02-05T06:19:28.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673018' 2024-02-05T06:19:28.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:28.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673018 -lt 42949673018 2024-02-05T06:19:28.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:28.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509496 2024-02-05T06:19:28.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:28.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:28.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509496 2024-02-05T06:19:28.957 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:28.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509496 2024-02-05T06:19:28.958 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509496 2024-02-05T06:19:28.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2024-02-05T06:19:28.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:29.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509496 -lt 64424509496 2024-02-05T06:19:29.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:29.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:29.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:29.473 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:29.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:29.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:29.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:29.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:29.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:29.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:29.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:29.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836542 2024-02-05T06:19:29.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836542 2024-02-05T06:19:29.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542' 2024-02-05T06:19:29.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:29.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:29.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673020 2024-02-05T06:19:29.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673020 2024-02-05T06:19:29.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-42949673020' 2024-02-05T06:19:29.987 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:29.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:30.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509497 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509497 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836542 1-42949673020 2-64424509497' 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836542 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836542 2024-02-05T06:19:30.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:30.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836542 2024-02-05T06:19:30.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-02-05T06:19:30.090 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836542 2024-02-05T06:19:30.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:30.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836542 2024-02-05T06:19:30.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:31.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:31.353 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:31.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836542 2024-02-05T06:19:31.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:31.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673020 2024-02-05T06:19:31.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:31.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:31.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673020 2024-02-05T06:19:31.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:31.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673020 2024-02-05T06:19:31.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-02-05T06:19:31.657 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673020 2024-02-05T06:19:31.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:31.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673020 -lt 42949673020 2024-02-05T06:19:31.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:31.924 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509497 2024-02-05T06:19:31.924 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:31.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:31.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509497 2024-02-05T06:19:31.925 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:31.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509497 2024-02-05T06:19:31.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2024-02-05T06:19:31.927 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509497 2024-02-05T06:19:31.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:32.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509497 2024-02-05T06:19:32.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:32.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:32.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:32.442 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:32.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:32.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:32.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:32.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:32.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:32.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:32.733 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:32.733 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:32.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:32.733 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:32.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:32.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-02-05T06:19:32.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-02-05T06:19:32.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-02-05T06:19:32.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:32.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673021 2024-02-05T06:19:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673021 2024-02-05T06:19:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673021' 2024-02-05T06:19:32.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:32.941 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:33.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509499 2024-02-05T06:19:33.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509499 2024-02-05T06:19:33.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-42949673021 2-64424509499' 2024-02-05T06:19:33.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:33.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-02-05T06:19:33.041 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:33.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:33.042 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-02-05T06:19:33.042 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:33.043 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836543 2024-02-05T06:19:33.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-02-05T06:19:33.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-02-05T06:19:33.044 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:33.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-02-05T06:19:33.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:34.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:34.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:34.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836543 2024-02-05T06:19:34.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:34.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673021 2024-02-05T06:19:34.584 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:34.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:34.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673021 2024-02-05T06:19:34.585 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:34.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673021 2024-02-05T06:19:34.586 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673021 2024-02-05T06:19:34.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673021' 2024-02-05T06:19:34.587 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:34.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673021 -lt 42949673021 2024-02-05T06:19:34.850 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:34.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509499 2024-02-05T06:19:34.850 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:34.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:34.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509499 2024-02-05T06:19:34.852 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:34.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509499 2024-02-05T06:19:34.853 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509499 2024-02-05T06:19:34.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509499' 2024-02-05T06:19:34.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:35.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509499 -lt 64424509499 2024-02-05T06:19:35.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:35.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:35.121 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:35.369 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:35.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:35.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:35.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:35.380 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:35.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:35.646 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:35.646 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:35.647 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:35.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:35.647 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:35.647 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:35.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836545 2024-02-05T06:19:35.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836545 2024-02-05T06:19:35.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545' 2024-02-05T06:19:35.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:35.747 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:35.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673023 2024-02-05T06:19:35.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673023 2024-02-05T06:19:35.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-42949673023' 2024-02-05T06:19:35.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:35.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:35.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509501 2024-02-05T06:19:35.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509501 2024-02-05T06:19:35.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836545 1-42949673023 2-64424509501' 2024-02-05T06:19:35.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:35.948 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836545 2024-02-05T06:19:35.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:35.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:35.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836545 2024-02-05T06:19:35.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:35.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836545 2024-02-05T06:19:35.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836545' 2024-02-05T06:19:35.951 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836545 2024-02-05T06:19:35.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:36.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836545 2024-02-05T06:19:36.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:37.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:37.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:37.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836545 2024-02-05T06:19:37.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:38.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:19:38.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:38.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836545 -lt 21474836545 2024-02-05T06:19:38.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:38.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673023 2024-02-05T06:19:38.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:38.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:38.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673023 2024-02-05T06:19:38.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:38.771 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673023 2024-02-05T06:19:38.771 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673023 2024-02-05T06:19:38.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-02-05T06:19:38.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:39.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673023 -lt 42949673023 2024-02-05T06:19:39.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:39.038 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509501 2024-02-05T06:19:39.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:39.039 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:39.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509501 2024-02-05T06:19:39.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:39.041 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509501 2024-02-05T06:19:39.041 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509501 2024-02-05T06:19:39.042 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509501' 2024-02-05T06:19:39.042 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:39.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509501 -lt 64424509501 2024-02-05T06:19:39.309 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:39.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:39.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:39.559 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:39.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:39.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:39.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:39.571 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:39.571 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:39.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:39.872 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:39.872 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:39.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:39.872 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:39.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:39.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836547 2024-02-05T06:19:39.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836547 2024-02-05T06:19:39.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547' 2024-02-05T06:19:39.973 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:39.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:40.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-02-05T06:19:40.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-02-05T06:19:40.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673025' 2024-02-05T06:19:40.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:40.074 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:40.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509502 2024-02-05T06:19:40.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509502 2024-02-05T06:19:40.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836547 1-42949673025 2-64424509502' 2024-02-05T06:19:40.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:40.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836547 2024-02-05T06:19:40.175 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:40.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:40.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836547 2024-02-05T06:19:40.176 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:40.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836547 2024-02-05T06:19:40.177 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836547 2024-02-05T06:19:40.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2024-02-05T06:19:40.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:40.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836546 -lt 21474836547 2024-02-05T06:19:40.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:41.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:41.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:41.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836547 2024-02-05T06:19:41.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:41.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-02-05T06:19:41.716 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:41.717 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:41.718 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-02-05T06:19:41.718 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:41.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-02-05T06:19:41.719 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673025 2024-02-05T06:19:41.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-02-05T06:19:41.719 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:41.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673025 2024-02-05T06:19:41.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:41.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509502 2024-02-05T06:19:41.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:41.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:41.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509502 2024-02-05T06:19:41.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:41.995 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509502 2024-02-05T06:19:41.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509502 2024-02-05T06:19:41.995 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-02-05T06:19:41.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:42.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509502 2024-02-05T06:19:42.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:42.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:42.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:42.511 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:42.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:42.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:42.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:42.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:42.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:42.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:42.814 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:42.814 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:42.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:42.814 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:42.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:42.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-02-05T06:19:42.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-02-05T06:19:42.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-02-05T06:19:42.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:42.972 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:43.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673026 2024-02-05T06:19:43.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673026 2024-02-05T06:19:43.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673026' 2024-02-05T06:19:43.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:43.072 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509504 2024-02-05T06:19:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509504 2024-02-05T06:19:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673026 2-64424509504' 2024-02-05T06:19:43.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:43.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-02-05T06:19:43.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:43.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:43.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-02-05T06:19:43.177 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:43.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-02-05T06:19:43.178 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836548 2024-02-05T06:19:43.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-02-05T06:19:43.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836547 -lt 21474836548 2024-02-05T06:19:43.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:44.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:44.459 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836548 2024-02-05T06:19:44.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:44.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673026 2024-02-05T06:19:44.731 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:44.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:44.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673026 2024-02-05T06:19:44.733 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:44.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673026 2024-02-05T06:19:44.734 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673026 2024-02-05T06:19:44.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-02-05T06:19:44.734 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:44.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673026 2024-02-05T06:19:44.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:45.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509504 2024-02-05T06:19:45.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:45.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:45.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509504 2024-02-05T06:19:45.001 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:45.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509504 2024-02-05T06:19:45.003 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509504 2024-02-05T06:19:45.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509504' 2024-02-05T06:19:45.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:45.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509504 -lt 64424509504 2024-02-05T06:19:45.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:45.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:45.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:45.521 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:45.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:45.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:45.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:45.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:45.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:45.824 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:45.827 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:45.828 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:45.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-02-05T06:19:45.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-02-05T06:19:45.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-02-05T06:19:45.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:45.926 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:46.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673028 2024-02-05T06:19:46.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673028 2024-02-05T06:19:46.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673028' 2024-02-05T06:19:46.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:46.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:46.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509506 2024-02-05T06:19:46.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509506 2024-02-05T06:19:46.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673028 2-64424509506' 2024-02-05T06:19:46.127 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:46.128 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-02-05T06:19:46.128 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:46.128 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:46.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-02-05T06:19:46.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:46.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-02-05T06:19:46.130 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836550 2024-02-05T06:19:46.131 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-02-05T06:19:46.131 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:46.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836550 2024-02-05T06:19:46.393 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:47.394 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:47.395 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:47.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-02-05T06:19:47.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:47.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673028 2024-02-05T06:19:47.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:47.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:47.673 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673028 2024-02-05T06:19:47.673 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:47.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673028 2024-02-05T06:19:47.674 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673028 2024-02-05T06:19:47.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673028' 2024-02-05T06:19:47.675 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:47.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673028 -lt 42949673028 2024-02-05T06:19:47.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:47.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509506 2024-02-05T06:19:47.940 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:47.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:47.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509506 2024-02-05T06:19:47.942 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:47.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509506 2024-02-05T06:19:47.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509506' 2024-02-05T06:19:47.943 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509506 2024-02-05T06:19:47.943 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:48.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509506 -lt 64424509506 2024-02-05T06:19:48.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:48.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:48.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:48.505 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:48.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:48.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:48.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:48.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:48.517 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:48.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:48.784 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:48.784 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:48.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:48.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:48.784 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:48.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836552 2024-02-05T06:19:48.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836552 2024-02-05T06:19:48.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552' 2024-02-05T06:19:48.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:48.890 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:48.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673029 2024-02-05T06:19:48.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673029 2024-02-05T06:19:48.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673029' 2024-02-05T06:19:48.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:48.992 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:49.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-02-05T06:19:49.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-02-05T06:19:49.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673029 2-64424509507' 2024-02-05T06:19:49.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:49.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836552 2024-02-05T06:19:49.093 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:49.093 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:49.094 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836552 2024-02-05T06:19:49.094 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:49.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836552 2024-02-05T06:19:49.095 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836552 2024-02-05T06:19:49.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-02-05T06:19:49.096 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:49.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836552 2024-02-05T06:19:49.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:50.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:50.360 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:50.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836552 2024-02-05T06:19:50.639 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:50.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673029 2024-02-05T06:19:50.640 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:50.640 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:50.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673029 2024-02-05T06:19:50.641 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:50.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673029 2024-02-05T06:19:50.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-02-05T06:19:50.642 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673029 2024-02-05T06:19:50.643 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:50.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673030 -lt 42949673029 2024-02-05T06:19:50.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:50.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-02-05T06:19:50.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:50.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:50.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-02-05T06:19:50.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:50.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-02-05T06:19:50.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-02-05T06:19:50.909 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509507 2024-02-05T06:19:50.910 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:51.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509507 2024-02-05T06:19:51.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:51.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:51.174 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:51.424 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:51.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:51.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:51.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:51.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:51.436 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:51.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:51.722 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:51.723 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:51.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:51.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:51.723 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:51.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836553 2024-02-05T06:19:51.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836553 2024-02-05T06:19:51.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553' 2024-02-05T06:19:51.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:51.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:51.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673031 2024-02-05T06:19:51.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673031 2024-02-05T06:19:51.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673031' 2024-02-05T06:19:51.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:51.924 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:52.024 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509509 2024-02-05T06:19:52.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509509 2024-02-05T06:19:52.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836553 1-42949673031 2-64424509509' 2024-02-05T06:19:52.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:52.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836553 2024-02-05T06:19:52.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:52.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:52.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836553 2024-02-05T06:19:52.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:52.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836553 2024-02-05T06:19:52.028 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836553 2024-02-05T06:19:52.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836553' 2024-02-05T06:19:52.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:52.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836553 2024-02-05T06:19:52.291 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:53.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:53.292 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:53.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836553 2024-02-05T06:19:53.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:54.590 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:19:54.590 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:54.861 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836553 2024-02-05T06:19:54.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:54.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673031 2024-02-05T06:19:54.862 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:54.863 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:54.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673031 2024-02-05T06:19:54.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:54.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673031 2024-02-05T06:19:54.865 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673031 2024-02-05T06:19:54.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673031' 2024-02-05T06:19:54.865 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:55.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673031 2024-02-05T06:19:55.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:55.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509509 2024-02-05T06:19:55.132 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:55.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:55.134 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509509 2024-02-05T06:19:55.134 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:55.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509509 2024-02-05T06:19:55.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509509' 2024-02-05T06:19:55.135 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509509 2024-02-05T06:19:55.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:55.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509509 -lt 64424509509 2024-02-05T06:19:55.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:55.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:55.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:55.650 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:55.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:55.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:55.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:55.661 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:55.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:55.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:55.922 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:55.922 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:55.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:55.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:55.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:56.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836555 2024-02-05T06:19:56.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836555 2024-02-05T06:19:56.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555' 2024-02-05T06:19:56.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:56.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:56.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-02-05T06:19:56.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-02-05T06:19:56.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673033' 2024-02-05T06:19:56.126 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:56.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:56.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-02-05T06:19:56.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-02-05T06:19:56.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836555 1-42949673033 2-64424509511' 2024-02-05T06:19:56.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:56.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836555 2024-02-05T06:19:56.227 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:56.227 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:56.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836555 2024-02-05T06:19:56.228 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:56.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836555 2024-02-05T06:19:56.230 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836555' 2024-02-05T06:19:56.230 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836555 2024-02-05T06:19:56.230 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:56.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836555 2024-02-05T06:19:56.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:19:57.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:19:57.496 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:57.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836555 2024-02-05T06:19:57.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:57.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-02-05T06:19:57.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:57.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:19:57.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-02-05T06:19:57.776 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:57.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-02-05T06:19:57.777 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-02-05T06:19:57.778 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673033 2024-02-05T06:19:57.778 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:19:58.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673033 -lt 42949673033 2024-02-05T06:19:58.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:58.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-02-05T06:19:58.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:58.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:19:58.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-02-05T06:19:58.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:58.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-02-05T06:19:58.061 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509511 2024-02-05T06:19:58.061 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-02-05T06:19:58.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:19:58.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509511 2024-02-05T06:19:58.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:19:58.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:19:58.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:19:58.577 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:19:58.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:19:58.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:19:58.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:19:58.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:19:58.588 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:19:58.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:19:58.854 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:19:58.854 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:19:58.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:19:58.854 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:58.854 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:19:58.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836557 2024-02-05T06:19:58.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836557 2024-02-05T06:19:58.954 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557' 2024-02-05T06:19:58.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:58.955 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:19:59.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673034 2024-02-05T06:19:59.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673034 2024-02-05T06:19:59.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673034' 2024-02-05T06:19:59.055 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:19:59.055 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:19:59.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509512 2024-02-05T06:19:59.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509512 2024-02-05T06:19:59.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836557 1-42949673034 2-64424509512' 2024-02-05T06:19:59.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:19:59.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836557 2024-02-05T06:19:59.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:19:59.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:19:59.157 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836557 2024-02-05T06:19:59.157 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:19:59.158 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836557 2024-02-05T06:19:59.158 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836557 2024-02-05T06:19:59.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836557' 2024-02-05T06:19:59.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:19:59.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836555 -lt 21474836557 2024-02-05T06:19:59.423 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:00.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:00.424 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:00.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836557 2024-02-05T06:20:00.690 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:00.690 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673034 2024-02-05T06:20:00.691 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:00.691 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:00.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673034 2024-02-05T06:20:00.692 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:00.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673034 2024-02-05T06:20:00.693 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673034 2024-02-05T06:20:00.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-02-05T06:20:00.694 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:00.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673034 2024-02-05T06:20:00.958 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:00.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509512 2024-02-05T06:20:00.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:00.959 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:00.959 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509512 2024-02-05T06:20:00.960 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:00.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509512 2024-02-05T06:20:00.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509512' 2024-02-05T06:20:00.961 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509512 2024-02-05T06:20:00.961 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:01.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509512 -lt 64424509512 2024-02-05T06:20:01.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:01.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:01.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:01.473 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:01.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:01.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:01.485 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:01.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:01.758 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:01.758 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:01.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:01.758 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:01.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:01.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836558 2024-02-05T06:20:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836558 2024-02-05T06:20:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558' 2024-02-05T06:20:01.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:01.887 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:01.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673036 2024-02-05T06:20:01.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673036 2024-02-05T06:20:01.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673036' 2024-02-05T06:20:01.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:01.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:02.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-02-05T06:20:02.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-02-05T06:20:02.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836558 1-42949673036 2-64424509514' 2024-02-05T06:20:02.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:02.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836558 2024-02-05T06:20:02.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:02.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:02.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836558 2024-02-05T06:20:02.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:02.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836558 2024-02-05T06:20:02.088 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836558 2024-02-05T06:20:02.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2024-02-05T06:20:02.089 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:02.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836557 -lt 21474836558 2024-02-05T06:20:02.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:03.354 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:03.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:03.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836558 2024-02-05T06:20:03.653 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:03.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673036 2024-02-05T06:20:03.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:03.655 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:03.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673036 2024-02-05T06:20:03.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:03.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673036 2024-02-05T06:20:03.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673036' 2024-02-05T06:20:03.657 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673036 2024-02-05T06:20:03.657 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:03.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673036 2024-02-05T06:20:03.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:03.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-02-05T06:20:03.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:03.924 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:03.924 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-02-05T06:20:03.924 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:03.925 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509514 2024-02-05T06:20:03.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-02-05T06:20:03.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-02-05T06:20:03.926 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:04.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509514 -lt 64424509514 2024-02-05T06:20:04.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:04.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:04.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:04.457 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:04.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:04.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:04.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:04.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:04.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:04.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:04.734 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:04.734 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:04.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:04.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:04.735 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:04.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836560 2024-02-05T06:20:04.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836560 2024-02-05T06:20:04.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560' 2024-02-05T06:20:04.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:04.835 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:04.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673038 2024-02-05T06:20:04.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673038 2024-02-05T06:20:04.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673038' 2024-02-05T06:20:04.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:04.937 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:05.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-02-05T06:20:05.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-02-05T06:20:05.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673038 2-64424509515' 2024-02-05T06:20:05.037 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:05.037 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836560 2024-02-05T06:20:05.037 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:05.038 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:05.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836560 2024-02-05T06:20:05.039 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:05.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836560 2024-02-05T06:20:05.040 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-02-05T06:20:05.040 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836560 2024-02-05T06:20:05.040 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:05.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836560 2024-02-05T06:20:05.303 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:06.304 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:06.304 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:06.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836560 2024-02-05T06:20:06.575 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:06.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673038 2024-02-05T06:20:06.576 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:06.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:06.577 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673038 2024-02-05T06:20:06.577 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:06.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673038 2024-02-05T06:20:06.578 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673038' 2024-02-05T06:20:06.579 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673038 2024-02-05T06:20:06.579 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:06.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673038 -lt 42949673038 2024-02-05T06:20:06.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:06.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-02-05T06:20:06.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:06.844 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:06.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-02-05T06:20:06.845 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:06.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-02-05T06:20:06.846 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509515 2024-02-05T06:20:06.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-02-05T06:20:06.847 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:07.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509515 2024-02-05T06:20:07.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:07.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:07.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:07.359 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:07.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:07.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:07.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:07.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:07.371 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:07.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:07.644 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:07.644 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:07.644 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:07.645 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:07.645 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836561 2024-02-05T06:20:07.745 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836561 2024-02-05T06:20:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561' 2024-02-05T06:20:07.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:07.746 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:07.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673039 2024-02-05T06:20:07.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673039 2024-02-05T06:20:07.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673039' 2024-02-05T06:20:07.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:07.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:07.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509517 2024-02-05T06:20:07.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509517 2024-02-05T06:20:07.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836561 1-42949673039 2-64424509517' 2024-02-05T06:20:07.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:07.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836561 2024-02-05T06:20:07.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:07.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:07.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836561 2024-02-05T06:20:07.982 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:07.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836561 2024-02-05T06:20:07.983 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836561 2024-02-05T06:20:07.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2024-02-05T06:20:07.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836561 2024-02-05T06:20:08.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:09.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:09.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:09.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836561 2024-02-05T06:20:09.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:10.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:20:10.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:10.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836561 2024-02-05T06:20:10.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:10.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673039 2024-02-05T06:20:10.798 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:10.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:10.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673039 2024-02-05T06:20:10.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:10.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673039 2024-02-05T06:20:10.800 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673039 2024-02-05T06:20:10.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-02-05T06:20:10.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:11.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673039 2024-02-05T06:20:11.066 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:11.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509517 2024-02-05T06:20:11.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:11.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:11.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509517 2024-02-05T06:20:11.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:11.069 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509517 2024-02-05T06:20:11.069 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509517 2024-02-05T06:20:11.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509517' 2024-02-05T06:20:11.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:11.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509517 -lt 64424509517 2024-02-05T06:20:11.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:11.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:11.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:11.586 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:11.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:11.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:11.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:11.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:11.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:11.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:11.880 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:11.880 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:11.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:11.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:11.880 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:11.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836563 2024-02-05T06:20:11.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836563 2024-02-05T06:20:11.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563' 2024-02-05T06:20:11.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:11.981 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:12.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-02-05T06:20:12.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-02-05T06:20:12.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673041' 2024-02-05T06:20:12.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:12.082 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:12.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-02-05T06:20:12.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-02-05T06:20:12.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673041 2-64424509519' 2024-02-05T06:20:12.182 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:12.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836563 2024-02-05T06:20:12.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:12.183 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:12.183 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836563 2024-02-05T06:20:12.184 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:12.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836563 2024-02-05T06:20:12.185 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836563 2024-02-05T06:20:12.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-02-05T06:20:12.185 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:12.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836563 2024-02-05T06:20:12.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:13.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:13.453 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:13.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836563 2024-02-05T06:20:13.728 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:13.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-02-05T06:20:13.729 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:13.729 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:13.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-02-05T06:20:13.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:13.731 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-02-05T06:20:13.731 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673041 2024-02-05T06:20:13.732 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-02-05T06:20:13.732 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:13.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-02-05T06:20:13.996 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:13.996 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-02-05T06:20:13.997 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:13.997 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:13.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-02-05T06:20:13.998 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:13.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-02-05T06:20:13.999 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509519 2024-02-05T06:20:13.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-02-05T06:20:13.999 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:14.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509519 -lt 64424509519 2024-02-05T06:20:14.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:14.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:14.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:14.530 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:14.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:14.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:14.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:14.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:14.542 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:14.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:14.821 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:14.821 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:14.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:14.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:14.821 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-02-05T06:20:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-02-05T06:20:14.922 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-02-05T06:20:14.923 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:14.923 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:15.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-02-05T06:20:15.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-02-05T06:20:15.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673043' 2024-02-05T06:20:15.023 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:15.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:15.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-02-05T06:20:15.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-02-05T06:20:15.122 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673043 2-64424509520' 2024-02-05T06:20:15.123 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:15.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-02-05T06:20:15.123 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:15.124 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:15.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-02-05T06:20:15.124 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:15.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-02-05T06:20:15.125 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-02-05T06:20:15.126 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836565 2024-02-05T06:20:15.126 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:15.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836565 2024-02-05T06:20:15.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:16.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:16.390 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:16.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836565 2024-02-05T06:20:16.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:16.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-02-05T06:20:16.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:16.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:16.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-02-05T06:20:16.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-02-05T06:20:16.669 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673043 2024-02-05T06:20:16.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-02-05T06:20:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:16.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673043 -lt 42949673043 2024-02-05T06:20:16.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:16.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-02-05T06:20:16.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:16.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:16.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-02-05T06:20:16.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:16.937 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-02-05T06:20:16.937 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509520 2024-02-05T06:20:16.938 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-02-05T06:20:16.938 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:17.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509520 2024-02-05T06:20:17.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:17.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:17.247 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:17.499 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:17.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:17.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:17.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:17.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:17.510 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:17.774 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:17.775 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:17.775 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:17.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:17.775 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:17.775 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:17.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836566 2024-02-05T06:20:17.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836566 2024-02-05T06:20:17.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566' 2024-02-05T06:20:17.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:17.904 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:18.003 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-02-05T06:20:18.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-02-05T06:20:18.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673044' 2024-02-05T06:20:18.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:18.004 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:18.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509522 2024-02-05T06:20:18.105 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509522 2024-02-05T06:20:18.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836566 1-42949673044 2-64424509522' 2024-02-05T06:20:18.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:18.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836566 2024-02-05T06:20:18.106 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:18.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:18.107 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836566 2024-02-05T06:20:18.108 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:18.108 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836566 2024-02-05T06:20:18.109 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836566 2024-02-05T06:20:18.109 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836566' 2024-02-05T06:20:18.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:18.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836566 2024-02-05T06:20:18.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:19.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:19.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:19.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836566 2024-02-05T06:20:19.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:19.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-02-05T06:20:19.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:19.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:19.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-02-05T06:20:19.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:19.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-02-05T06:20:19.666 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673044 2024-02-05T06:20:19.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-02-05T06:20:19.667 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:19.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673044 2024-02-05T06:20:19.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:19.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509522 2024-02-05T06:20:19.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:19.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:19.933 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509522 2024-02-05T06:20:19.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:19.934 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509522 2024-02-05T06:20:19.935 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509522 2024-02-05T06:20:19.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-02-05T06:20:19.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:20.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509522 2024-02-05T06:20:20.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:20.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:20.213 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:20.463 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:20.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:20.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:20.474 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:20.475 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:20.475 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:20.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:20.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:20.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-02-05T06:20:20.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-02-05T06:20:20.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-02-05T06:20:20.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:20.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:20.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-02-05T06:20:20.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-02-05T06:20:20.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046' 2024-02-05T06:20:20.975 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:20.975 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:21.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509524 2024-02-05T06:20:21.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509524 2024-02-05T06:20:21.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673046 2-64424509524' 2024-02-05T06:20:21.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:21.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-02-05T06:20:21.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:21.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:21.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-02-05T06:20:21.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:21.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-02-05T06:20:21.079 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836568 2024-02-05T06:20:21.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-02-05T06:20:21.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:21.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836568 2024-02-05T06:20:21.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:22.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:22.342 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:22.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836568 2024-02-05T06:20:22.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:22.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-02-05T06:20:22.617 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:22.617 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:22.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-02-05T06:20:22.618 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:22.619 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673046 2024-02-05T06:20:22.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-02-05T06:20:22.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-02-05T06:20:22.620 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:22.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-02-05T06:20:22.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:22.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509524 2024-02-05T06:20:22.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:22.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:22.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509524 2024-02-05T06:20:22.885 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:22.886 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509524 2024-02-05T06:20:22.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509524 2024-02-05T06:20:22.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-02-05T06:20:22.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:23.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509524 -lt 64424509524 2024-02-05T06:20:23.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:23.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:23.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:23.403 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:23.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:23.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:23.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:23.414 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:23.415 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:23.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:23.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:23.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836570 2024-02-05T06:20:23.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836570 2024-02-05T06:20:23.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570' 2024-02-05T06:20:23.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:23.781 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:23.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673047 2024-02-05T06:20:23.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673047 2024-02-05T06:20:23.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673047' 2024-02-05T06:20:23.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:23.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:23.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509525 2024-02-05T06:20:23.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509525 2024-02-05T06:20:23.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509525' 2024-02-05T06:20:23.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:23.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836570 2024-02-05T06:20:23.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:23.983 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:23.984 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836570 2024-02-05T06:20:23.984 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:23.985 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836570 2024-02-05T06:20:23.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836570 2024-02-05T06:20:23.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-02-05T06:20:23.985 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:24.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836570 2024-02-05T06:20:24.250 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:25.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:25.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:25.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836570 2024-02-05T06:20:25.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:26.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:20:26.532 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:26.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836570 2024-02-05T06:20:26.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:26.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673047 2024-02-05T06:20:26.808 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:26.809 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:26.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673047 2024-02-05T06:20:26.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:26.810 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673047 2024-02-05T06:20:26.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673047 2024-02-05T06:20:26.811 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-02-05T06:20:26.811 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:27.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673048 -lt 42949673047 2024-02-05T06:20:27.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:27.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509525 2024-02-05T06:20:27.077 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:27.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:27.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509525 2024-02-05T06:20:27.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:27.079 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509525 2024-02-05T06:20:27.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509525 2024-02-05T06:20:27.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509525' 2024-02-05T06:20:27.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:27.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509525 2024-02-05T06:20:27.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:27.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:27.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:27.598 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:27.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:27.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:27.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:27.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:27.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:27.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:27.929 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:27.929 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:27.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:27.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:27.929 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:27.988 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836571 2024-02-05T06:20:27.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836571 2024-02-05T06:20:27.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571' 2024-02-05T06:20:27.989 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:27.989 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:28.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-02-05T06:20:28.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-02-05T06:20:28.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673049' 2024-02-05T06:20:28.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:28.090 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:28.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-02-05T06:20:28.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-02-05T06:20:28.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836571 1-42949673049 2-64424509527' 2024-02-05T06:20:28.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:28.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836571 2024-02-05T06:20:28.190 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:28.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:28.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836571 2024-02-05T06:20:28.191 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:28.192 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836571 2024-02-05T06:20:28.192 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836571 2024-02-05T06:20:28.193 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836571' 2024-02-05T06:20:28.193 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:28.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836570 -lt 21474836571 2024-02-05T06:20:28.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:29.457 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:29.458 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:29.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836571 2024-02-05T06:20:29.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:29.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-02-05T06:20:29.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:29.770 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:29.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-02-05T06:20:29.770 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:29.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-02-05T06:20:29.772 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-02-05T06:20:29.772 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673049 2024-02-05T06:20:29.772 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:30.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673050 -lt 42949673049 2024-02-05T06:20:30.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:30.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-02-05T06:20:30.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:30.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:30.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-02-05T06:20:30.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:30.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-02-05T06:20:30.060 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509527 2024-02-05T06:20:30.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-02-05T06:20:30.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509527 2024-02-05T06:20:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:30.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:30.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:30.576 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:30.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:30.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:30.588 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:30.589 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:30.589 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:30.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:30.851 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:30.851 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:30.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:30.851 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:30.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:30.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836573 2024-02-05T06:20:30.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836573 2024-02-05T06:20:30.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573' 2024-02-05T06:20:30.952 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:30.952 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:31.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-02-05T06:20:31.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-02-05T06:20:31.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673051' 2024-02-05T06:20:31.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:31.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:31.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509529 2024-02-05T06:20:31.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509529 2024-02-05T06:20:31.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673051 2-64424509529' 2024-02-05T06:20:31.154 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:31.154 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836573 2024-02-05T06:20:31.155 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:31.155 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:31.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836573 2024-02-05T06:20:31.156 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:31.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836573 2024-02-05T06:20:31.157 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836573 2024-02-05T06:20:31.157 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-02-05T06:20:31.157 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:31.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836573 2024-02-05T06:20:31.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:32.421 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:32.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:32.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836573 2024-02-05T06:20:32.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:32.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-02-05T06:20:32.693 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:32.694 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:32.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-02-05T06:20:32.695 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:32.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-02-05T06:20:32.696 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-02-05T06:20:32.696 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673051 2024-02-05T06:20:32.697 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:32.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 42949673051 2024-02-05T06:20:32.961 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:32.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509529 2024-02-05T06:20:32.962 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:32.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:32.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509529 2024-02-05T06:20:32.963 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:32.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509529 2024-02-05T06:20:32.965 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509529' 2024-02-05T06:20:32.965 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509529 2024-02-05T06:20:32.965 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:33.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509529 -lt 64424509529 2024-02-05T06:20:33.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:33.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:33.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:33.481 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:33.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:33.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:33.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:33.493 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:33.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:33.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:33.767 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:33.767 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:33.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:33.767 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:33.767 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:33.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-02-05T06:20:33.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-02-05T06:20:33.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-02-05T06:20:33.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:33.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-02-05T06:20:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-02-05T06:20:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673052' 2024-02-05T06:20:34.036 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:34.036 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:34.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509530 2024-02-05T06:20:34.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509530 2024-02-05T06:20:34.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673052 2-64424509530' 2024-02-05T06:20:34.137 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:34.137 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-02-05T06:20:34.138 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:34.138 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:34.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-02-05T06:20:34.139 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:34.140 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836575 2024-02-05T06:20:34.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-02-05T06:20:34.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-02-05T06:20:34.140 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:34.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836574 -lt 21474836575 2024-02-05T06:20:34.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:35.403 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:35.403 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:35.678 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836575 2024-02-05T06:20:35.679 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:35.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-02-05T06:20:35.679 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:35.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:35.680 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-02-05T06:20:35.681 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:35.681 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-02-05T06:20:35.682 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-02-05T06:20:35.682 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673052 2024-02-05T06:20:35.682 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:35.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673053 -lt 42949673052 2024-02-05T06:20:35.946 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:35.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509530 2024-02-05T06:20:35.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:35.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:35.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509530 2024-02-05T06:20:35.947 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:35.948 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509530 2024-02-05T06:20:35.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509530' 2024-02-05T06:20:35.949 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509530 2024-02-05T06:20:35.949 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:36.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509531 -lt 64424509530 2024-02-05T06:20:36.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:36.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:36.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:36.465 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:36.476 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:36.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:36.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:36.477 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:36.477 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:36.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:36.741 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:36.741 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:36.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:36.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:36.741 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:36.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836576 2024-02-05T06:20:36.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836576 2024-02-05T06:20:36.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576' 2024-02-05T06:20:36.866 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:36.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:36.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-02-05T06:20:36.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-02-05T06:20:36.966 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673054' 2024-02-05T06:20:36.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:36.967 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-02-05T06:20:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-02-05T06:20:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836576 1-42949673054 2-64424509532' 2024-02-05T06:20:37.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:37.067 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836576 2024-02-05T06:20:37.068 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:37.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:37.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836576 2024-02-05T06:20:37.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:37.070 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836576 2024-02-05T06:20:37.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836576 2024-02-05T06:20:37.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2024-02-05T06:20:37.070 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:37.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836576 2024-02-05T06:20:37.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:38.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:38.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:38.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836576 -lt 21474836576 2024-02-05T06:20:38.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:38.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-02-05T06:20:38.608 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:38.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:38.609 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-02-05T06:20:38.610 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:38.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-02-05T06:20:38.611 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673054 2024-02-05T06:20:38.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-02-05T06:20:38.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:38.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673054 -lt 42949673054 2024-02-05T06:20:38.875 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:38.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-02-05T06:20:38.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:38.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:38.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-02-05T06:20:38.877 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:38.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-02-05T06:20:38.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-02-05T06:20:38.879 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509532 2024-02-05T06:20:38.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:39.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-02-05T06:20:39.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:39.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:39.145 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:39.452 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:39.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:39.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:39.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:39.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:39.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:39.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:39.730 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:39.730 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:39.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:39.730 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:39.730 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:39.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-02-05T06:20:39.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-02-05T06:20:39.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-02-05T06:20:39.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:39.831 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673056 2024-02-05T06:20:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673056 2024-02-05T06:20:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673056' 2024-02-05T06:20:39.931 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:39.931 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:40.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-02-05T06:20:40.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-02-05T06:20:40.030 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673056 2-64424509533' 2024-02-05T06:20:40.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:40.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-02-05T06:20:40.031 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:40.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:40.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-02-05T06:20:40.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:40.033 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-02-05T06:20:40.033 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836578 2024-02-05T06:20:40.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-02-05T06:20:40.034 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:40.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836578 2024-02-05T06:20:40.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:41.298 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:41.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:41.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836578 2024-02-05T06:20:41.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:42.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-02-05T06:20:42.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:42.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836578 -lt 21474836578 2024-02-05T06:20:42.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:42.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673056 2024-02-05T06:20:42.872 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:42.873 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:42.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673056 2024-02-05T06:20:42.873 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:42.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673056 2024-02-05T06:20:42.874 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2024-02-05T06:20:42.875 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673056 2024-02-05T06:20:42.875 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:43.140 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673056 2024-02-05T06:20:43.141 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:43.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-02-05T06:20:43.141 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:43.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:43.142 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-02-05T06:20:43.143 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:43.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-02-05T06:20:43.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-02-05T06:20:43.144 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509533 2024-02-05T06:20:43.144 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509533 2024-02-05T06:20:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:43.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:43.659 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:43.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: continue 2024-02-05T06:20:43.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: for i in $(seq 0 200) 2024-02-05T06:20:43.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:377: _scrub_abort: flush_pg_stats 2024-02-05T06:20:43.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-02-05T06:20:43.671 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:20:43.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:20:43.935 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:20:43.935 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:20:43.935 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:20:43.936 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:43.936 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:20:44.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836580 2024-02-05T06:20:44.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836580 2024-02-05T06:20:44.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580' 2024-02-05T06:20:44.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:44.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:20:44.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673057 2024-02-05T06:20:44.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673057 2024-02-05T06:20:44.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673057' 2024-02-05T06:20:44.159 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:20:44.159 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:20:44.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509535 2024-02-05T06:20:44.259 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509535 2024-02-05T06:20:44.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836580 1-42949673057 2-64424509535' 2024-02-05T06:20:44.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:44.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836580 2024-02-05T06:20:44.260 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:44.261 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:20:44.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836580 2024-02-05T06:20:44.261 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:44.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836580 2024-02-05T06:20:44.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836580' 2024-02-05T06:20:44.263 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836580 2024-02-05T06:20:44.263 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:44.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836580 2024-02-05T06:20:44.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:20:45.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-02-05T06:20:45.530 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:20:45.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836580 -lt 21474836580 2024-02-05T06:20:45.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:45.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673057 2024-02-05T06:20:45.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:45.808 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:20:45.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673057 2024-02-05T06:20:45.809 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:45.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673057 2024-02-05T06:20:45.810 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949673057 2024-02-05T06:20:45.810 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-02-05T06:20:45.810 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:20:46.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673057 2024-02-05T06:20:46.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:20:46.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509535 2024-02-05T06:20:46.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:20:46.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:20:46.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509535 2024-02-05T06:20:46.081 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:20:46.082 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509535 2024-02-05T06:20:46.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509535 2024-02-05T06:20:46.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-02-05T06:20:46.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:20:46.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509536 -lt 64424509535 2024-02-05T06:20:46.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: ceph pg dump pgs 2024-02-05T06:20:46.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep '^1.0' 2024-02-05T06:20:46.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: grep -q scrubbing 2024-02-05T06:20:46.599 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:20:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:383: _scrub_abort: break 2024-02-05T06:20:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:385: _scrub_abort: set +o pipefail 2024-02-05T06:20:46.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:387: _scrub_abort: sleep 5 2024-02-05T06:20:51.611 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:389: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2024-02-05T06:20:51.626 INFO:tasks.workunit.client.0.smithi188.stdout:2024-02-05T06:18:47.617+0000 7f731d3e6640 10 osd.1 pg_epoch: 21 pg[1.0( v 19'1000 (0'0,19'1000] local-lis/les=17/18 n=1000 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) [1,0,2] r=0 lpr=17 crt=19'1000 lcod 19'999 mlcod 19'999 active+clean+scrubbing+deep [ 1.0: ] TIME_FOR_DEEP] scrubber: nodeep_scrub set, aborting 2024-02-05T06:20:51.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:395: _scrub_abort: get_last_scrub_stamp 1.0 2024-02-05T06:20:51.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:20:51.627 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:20:51.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:20:51.628 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:20:51.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:395: _scrub_abort: local last_scrub=2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:51.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:396: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2024-02-05T06:20:52.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: ceph osd unset nodeep-scrub 2024-02-05T06:20:52.596 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is unset 2024-02-05T06:20:52.607 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2024-02-05T06:20:52.608 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:401: _scrub_abort: ceph osd unset noscrub 2024-02-05T06:20:52.974 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is unset 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:403: _scrub_abort: TIMEOUT=500 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:404: _scrub_abort: wait_for_scrub 1.0 2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:20:52.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:20:53.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:53.248 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:20:54.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:20:54.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:20:54.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:20:54.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:20:54.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:20:54.250 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:20:54.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:20:54.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:54.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:20:55.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:20:55.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:20:55.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:20:55.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:20:55.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:20:55.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:20:55.522 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:20:55.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:55.787 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:20:56.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:20:56.788 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:20:56.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:20:56.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:20:56.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:20:56.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:20:56.789 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:20:57.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:57.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:20:58.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:20:58.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:20:58.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:20:58.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:20:58.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:20:58.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:20:58.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:20:58.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:58.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:20:59.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:20:59.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:20:59.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:20:59.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:20:59.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:20:59.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:20:59.327 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:20:59.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:20:59.596 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:00.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:00.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:00.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:00.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:00.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:00.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:00.598 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:00.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:00.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:01.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:01.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:01.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:01.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:01.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:01.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:01.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:02.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:02.167 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:03.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:03.168 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:03.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:03.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:03.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:03.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:03.169 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:03.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:03.439 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:04.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:04.440 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:04.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:04.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:04.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:04.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:04.441 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:04.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:04.709 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:05.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:05.710 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:05.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:05.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:05.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:05.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:05.711 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:05.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:05.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:06.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:06.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:06.986 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:06.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:06.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:06.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:06.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:07.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:07.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:08.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:08.258 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:08.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:08.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:08.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:08.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:08.259 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:08.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:08.519 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:09.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:09.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:09.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:09.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:09.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:09.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:09.521 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:09.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:09.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:10.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:10.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:10.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:10.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:10.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:10.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:10.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:11.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:11.067 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:12.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:12.068 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:12.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:12.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:12.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:12.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:12.069 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:12.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:12.341 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:13.342 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:13.343 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:13.343 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:13.343 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:13.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:13.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:13.344 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:13.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:13.609 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:14.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:14.610 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:14.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:14.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:14.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:14.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:14.611 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:14.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:14.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:15.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:15.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:15.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:15.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:15.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:15.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:15.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:16.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:16.171 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:17.172 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:17.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:17.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:17.173 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:17.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:17.174 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:17.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:17.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:18.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:18.443 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:18.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:18.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:18.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:18.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:18.444 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:18.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:18.711 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:19.712 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:19.713 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:19.713 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:19.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:19.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:19.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:19.714 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:19.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:19.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:20.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:20.986 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:20.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:20.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:20.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:20.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:20.987 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:21.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:21.256 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:22.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:22.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:22.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:22.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:22.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:22.258 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:22.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:22.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:23.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:23.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:23.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:23.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:23.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:23.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:23.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:23.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:23.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:24.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:24.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:24.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:24.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:24.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:24.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:24.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:25.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:25.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:26.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:26.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:26.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:26.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:26.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:26.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:26.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:26.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:27.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:27.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:27.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:27.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:27.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:27.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:27.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:27.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:27.622 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:28.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:28.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:28.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:28.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:28.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:28.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:28.624 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:28.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:28.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:29.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:29.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:29.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:29.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:29.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:29.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:29.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:30.178 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:30.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:31.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:31.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:31.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:31.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:31.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:31.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:31.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:31.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:31.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:32.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:32.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:32.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:32.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:32.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:32.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:32.454 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:32.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:32.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:33.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:33.727 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:33.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:33.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:33.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:33.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:33.728 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:33.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:33.992 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:34.993 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:34.994 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:34.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:34.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:34.994 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:34.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:34.995 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:35.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:35.262 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:36.263 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:36.264 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:36.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:36.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:37.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:37.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:37.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:37.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:37.533 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:37.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:37.534 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:37.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:37.805 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:38.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:38.806 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:38.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:38.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:38.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:38.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:38.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:39.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:39.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:40.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:40.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:40.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:40.078 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:40.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:40.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:40.079 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:40.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:41.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:41.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:41.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:41.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:41.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:41.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:41.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:41.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:41.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:42.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:42.620 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:42.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:42.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:42.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:42.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:42.621 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:42.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:42.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:43.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:43.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:43.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:43.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:43.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:43.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:43.911 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:44.179 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:44.180 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:45.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:45.181 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:45.181 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:45.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:45.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:45.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:45.182 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:45.453 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:45.454 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:46.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:46.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:46.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:46.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:46.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:46.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:46.456 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:46.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:46.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:47.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:47.725 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:47.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:47.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:47.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:47.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:47.726 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:48.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:48.001 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:49.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:49.002 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:49.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:49.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:49.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:49.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:49.003 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:49.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:49.271 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:50.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:50.272 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:50.273 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:50.273 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:50.273 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:50.273 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:50.273 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:50.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:50.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:51.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:51.543 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:51.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:51.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:51.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:51.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:51.545 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:51.815 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:51.816 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:52.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:52.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:52.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:52.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:52.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:52.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:52.818 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:53.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:53.094 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:54.095 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:54.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:54.096 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:54.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:54.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:54.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:54.097 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:54.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:54.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:55.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:55.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:55.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:55.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:55.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:55.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:55.370 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:55.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:55.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:56.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:56.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:56.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:56.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:56.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:56.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:56.644 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:56.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:56.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:57.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:57.918 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:57.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:57.918 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:57.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:57.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:57.919 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:58.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:58.194 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:21:59.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:21:59.195 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:21:59.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:21:59.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:21:59.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:21:59.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:21:59.197 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:21:59.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:21:59.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:00.472 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:00.473 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:00.473 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:00.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:00.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:00.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:00.474 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:00.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:00.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:01.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:01.744 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:01.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:01.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:01.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:01.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:01.745 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:02.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:02.017 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:03.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:03.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:03.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:03.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:03.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:03.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:03.019 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:03.284 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:03.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:04.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:04.285 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:04.286 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:04.286 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:04.287 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:04.287 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:04.287 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:04.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:04.555 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:05.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:05.556 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:05.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:05.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:05.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:05.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:05.557 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:05.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:05.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:06.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:06.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:06.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:06.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:06.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:06.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:06.830 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:07.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:07.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:08.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:08.101 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:08.365 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:09.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:09.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:09.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:09.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:09.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:09.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:09.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:09.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:09.637 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:10.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:10.638 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:10.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:10.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:10.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:10.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:10.639 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:10.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:11.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:11.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:11.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:11.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:11.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:11.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:11.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:12.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:12.176 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:13.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:13.177 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:13.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:13.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:13.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:13.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:13.178 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:13.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:13.441 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:14.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:14.442 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:14.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:14.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:14.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:14.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:14.443 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:14.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:14.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:15.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:15.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:15.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:15.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:15.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:15.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:15.708 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:15.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:15.981 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:16.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:16.982 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:16.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:16.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:16.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:16.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:16.983 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:17.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:17.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:18.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:18.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:18.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:18.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:18.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:18.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:18.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:18.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:18.517 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:19.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:19.518 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:19.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:19.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:19.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:19.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:19.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:19.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:19.791 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:20.792 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:20.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:20.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:20.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:20.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:20.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:20.866 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:21.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:21.062 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:22.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:22.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:22.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:22.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:22.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:22.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:22.064 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:22.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:22.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:23.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:23.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:23.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:23.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:23.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:23.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:23.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:23.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:23.603 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:24.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:24.604 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:24.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:24.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:24.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:24.605 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:24.606 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:24.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:24.876 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:25.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:25.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:25.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:25.878 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:25.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:25.879 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:26.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:26.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:27.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:27.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:27.149 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:27.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:27.419 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:28.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:28.420 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:28.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:28.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:28.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:28.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:28.421 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:28.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:28.686 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:29.688 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:29.689 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:29.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:29.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:29.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:29.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:29.689 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:29.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:29.956 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:30.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:30.957 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:30.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:30.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:30.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:30.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:30.958 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:31.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:31.224 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:32.225 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:32.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:32.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:32.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:32.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:32.226 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:32.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:32.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:33.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:33.496 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:33.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:33.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:33.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:33.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:33.497 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:33.783 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:33.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:34.784 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:34.785 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:34.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:34.785 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:34.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:34.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:34.786 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:35.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:35.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:36.059 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:36.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:36.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:36.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:36.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:36.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:36.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:36.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:36.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:37.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:37.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:37.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:37.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:37.334 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:37.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:37.335 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:37.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:37.605 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:38.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:38.606 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:38.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:38.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:38.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:38.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:38.607 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:38.912 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:38.913 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:39.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:39.916 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:39.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:39.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:39.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:39.916 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:39.917 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:40.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:40.184 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:41.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:41.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:41.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:41.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:41.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:41.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:41.186 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:41.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:41.458 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:42.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:42.459 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:42.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:42.460 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:42.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:42.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:42.461 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:42.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:42.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:43.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:43.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:43.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:43.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:43.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:43.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:43.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:44.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:44.007 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:45.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:45.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:45.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:45.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:45.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:45.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:45.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:45.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:45.276 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:46.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:46.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:46.278 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:46.278 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:46.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:46.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:46.279 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:46.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:46.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:47.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:47.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:47.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:47.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:47.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:47.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:47.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:47.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:47.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:48.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:48.821 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:48.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:48.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:48.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:48.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:48.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:49.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:49.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:50.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:50.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:50.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:50.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:50.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:50.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:50.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:50.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:50.357 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:51.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:51.358 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:51.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:51.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:51.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:51.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:51.359 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:51.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:51.630 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:52.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:52.631 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:52.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:52.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:52.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:52.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:52.632 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:52.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:52.925 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:53.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:53.926 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:53.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:53.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:53.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:53.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:53.927 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:54.189 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:54.190 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:55.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:55.191 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:55.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:55.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:55.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:55.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:55.192 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:55.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:55.461 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:56.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:56.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:56.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:56.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:56.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:56.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:56.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:56.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:56.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:57.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:57.736 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:57.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:57.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:57.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:57.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:57.737 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:58.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:58.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:22:59.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:22:59.009 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:22:59.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:22:59.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:22:59.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:22:59.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:22:59.010 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:22:59.277 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:22:59.278 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:00.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:00.279 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:00.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:00.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:00.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:00.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:00.280 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:00.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:00.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:01.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:01.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:01.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:01.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:01.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:01.550 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:01.551 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:01.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:01.818 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:02.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:02.819 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:02.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:02.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:02.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:02.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:02.820 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:03.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:03.089 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:04.090 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:04.091 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:04.091 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:04.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:04.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:04.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:04.092 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:04.359 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:04.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:05.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:05.360 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:05.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:05.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:05.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:05.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:05.361 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:05.634 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:05.635 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:06.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:06.636 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:06.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:06.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:06.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:06.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:06.637 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:06.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:06.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:07.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:07.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:07.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:07.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:07.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:07.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:07.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:08.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:08.202 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:09.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:09.205 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:09.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:09.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:09.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:09.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:09.205 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:09.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:10.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:10.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:10.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:10.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:10.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:10.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:10.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:10.734 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:11.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:11.735 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:11.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:11.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:11.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:11.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:11.736 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:12.004 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:12.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:13.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:13.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:13.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:13.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:13.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:13.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:13.006 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:13.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:13.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:14.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:14.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:14.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:14.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:14.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:14.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:14.271 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:14.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:14.537 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:15.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:15.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:15.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:15.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:15.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:15.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:15.539 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:15.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:15.812 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:16.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:16.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:16.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:16.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:16.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:16.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:16.814 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:17.080 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:17.081 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:18.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:18.082 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:18.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:18.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:18.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:18.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:18.083 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:18.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:18.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:19.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:19.353 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:19.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:19.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:19.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:19.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:19.354 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:19.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:19.628 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:20.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:20.629 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:20.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:20.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:20.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:20.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:20.630 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:20.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:20.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:21.933 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:21.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:21.934 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:21.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:21.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:21.935 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:22.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:22.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:23.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:23.210 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:23.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:23.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:23.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:23.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:23.211 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:23.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:23.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:24.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:24.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:24.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:24.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:24.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:24.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:24.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:24.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:24.754 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:25.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:25.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:25.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:25.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:25.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:25.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:25.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:26.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:26.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:27.026 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:27.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:27.027 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:27.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:27.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:27.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:27.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:27.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:27.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:28.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:28.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:28.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:28.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:28.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:28.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:28.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:28.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:28.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:29.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:29.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:29.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:29.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:29.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:29.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:29.567 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:29.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:29.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:30.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:30.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:30.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:30.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:30.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:30.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:30.841 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:31.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:31.111 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:32.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:32.112 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:32.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:32.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:32.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:32.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:32.113 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:32.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:32.383 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:33.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:33.384 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:33.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:33.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:33.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:33.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:33.385 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:33.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:33.656 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:34.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:34.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:34.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:34.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:34.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:34.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:34.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:34.927 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:34.928 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:35.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:35.929 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:35.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:35.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:35.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:35.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:35.930 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:36.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:36.200 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:37.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:37.199 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:37.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:37.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:37.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:37.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:37.200 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:37.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:37.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:38.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:38.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:38.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:38.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:38.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:38.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:38.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:38.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:38.740 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:39.742 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:39.743 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:39.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:39.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:39.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:39.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:39.743 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:40.018 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:40.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:41.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:41.020 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:41.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:41.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:41.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:41.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:41.021 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:41.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:41.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:42.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:42.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:42.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:42.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:42.294 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:42.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:42.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:42.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:42.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:43.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:43.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:43.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:43.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:43.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:43.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:43.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:43.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:43.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:44.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:44.836 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:44.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:44.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:44.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:44.837 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:44.838 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:45.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:45.106 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:46.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:46.107 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:46.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:46.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:46.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:46.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:46.109 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:46.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:46.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:47.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:47.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:47.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:47.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:47.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:47.380 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:47.381 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:47.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:47.650 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:48.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:48.652 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:48.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:48.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:48.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:48.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:48.653 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:48.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:48.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:49.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:49.943 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:49.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:49.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:49.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:49.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:49.944 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:50.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:50.208 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:51.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:51.209 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:51.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:51.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:51.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:51.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:51.210 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:51.479 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:51.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:52.480 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:52.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:52.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:52.481 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:52.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:52.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:52.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:52.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:52.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:53.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:53.752 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:53.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:53.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:53.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:53.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:53.753 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:54.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:54.021 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:55.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:55.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:55.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:55.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:55.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:55.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:55.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:55.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:55.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:56.294 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:56.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:56.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:56.295 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:56.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:56.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:56.296 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:56.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:56.562 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:57.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:57.564 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:57.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:57.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:57.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:57.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:57.565 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:57.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:57.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:23:58.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:23:58.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:23:58.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:23:58.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:23:58.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:23:58.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:23:58.839 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:23:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:23:59.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:00.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:00.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:00.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:00.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:00.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:00.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:00.115 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:00.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:00.387 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:01.388 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:01.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:01.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:01.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:01.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:01.389 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:01.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:01.658 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:02.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:02.659 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:02.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:02.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:02.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:02.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:02.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:02.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:02.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:03.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:04.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:04.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:04.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:04.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:04.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:04.246 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:04.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:04.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:05.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:05.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:05.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:05.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:05.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:05.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:05.216 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:05.484 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:05.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:06.485 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:06.486 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:06.486 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:06.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:06.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:06.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:06.487 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:07.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:07.757 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:07.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:07.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:07.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:07.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:07.759 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:08.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:08.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:09.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:09.029 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:09.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:09.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:09.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:09.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:09.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:09.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:09.307 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:10.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:10.308 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:10.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:10.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:10.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:10.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:10.309 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:10.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:10.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:11.580 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:11.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:11.581 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:11.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:11.846 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:12.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:12.847 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:12.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:12.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:12.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:12.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:12.848 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:13.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:13.119 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:14.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:14.120 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:14.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:14.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:14.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:14.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:14.121 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:14.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:14.395 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:15.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:15.396 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:15.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:15.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:15.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:15.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:15.397 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:15.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:15.667 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:16.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:16.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:16.668 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:16.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:16.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:16.944 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:17.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:17.945 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:17.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:17.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:17.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:17.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:17.946 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:18.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:18.217 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:19.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:19.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:19.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:19.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:19.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:19.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:19.219 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:19.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:19.490 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:20.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:20.491 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:20.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:20.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:20.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:20.492 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:20.493 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:20.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:20.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:21.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:21.756 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:21.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:21.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:21.757 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:21.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:21.758 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:22.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:22.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:23.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:23.027 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:23.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:23.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:23.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:23.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:23.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:23.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:23.296 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:24.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:24.297 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:24.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:24.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:24.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:24.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:24.299 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:24.569 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:24.570 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:25.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:25.568 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:25.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:25.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:25.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:25.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:25.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:25.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:25.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:26.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:26.843 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:26.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:26.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:26.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:26.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:26.844 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:27.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:27.115 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:28.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:28.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:28.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:28.117 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:28.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:28.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:28.118 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:28.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:28.389 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:29.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:29.390 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:29.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:29.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:29.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:29.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:29.391 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:29.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:30.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:30.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:30.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:30.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:30.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:30.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:30.666 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:30.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:30.968 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:31.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:31.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:31.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:31.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:31.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:31.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:31.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:32.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:32.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:33.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:33.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:33.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:33.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:33.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:33.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:33.242 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:33.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:33.509 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:34.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:34.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:34.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:34.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:34.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:34.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:34.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:34.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-01-22T06:16:59.961518+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:34.781 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-02-05T06:24:35.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-02-05T06:24:35.782 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 500 )) 2024-02-05T06:24:35.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-02-05T06:24:35.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-02-05T06:24:35.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-02-05T06:24:35.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-02-05T06:24:35.783 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-02-05T06:24:36.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-02-05T06:24:29.353929+0000 '>' 2024-01-22T06:16:59.961518+0000 2024-02-05T06:24:36.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-02-05T06:24:36.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: _scrub_abort: perf_counters td/osd-scrub-test 3 2024-02-05T06:24:36.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-02-05T06:24:36.052 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-02-05T06:24:36.053 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-02-05T06:24:36.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-02-05T06:24:36.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:24:36.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-02-05T06:24:36.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:24:36.147 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:24:36.147 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:24:36.147 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.147 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.147 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:24:36.147 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.148 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.149 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.150 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.151 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.152 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.153 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:24:36.154 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.155 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.156 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.157 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.158 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.159 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.160 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-02-05T06:24:36.161 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:24:36.247 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:24:36.247 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:24:36.247 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.247 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.247 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.248 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.249 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.250 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 2, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 2, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 1, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 1, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 207.090254067, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 207.090254067 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 1, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 1, 2024-02-05T06:24:36.251 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 5.000909183, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 5.000909183 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 41, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 2, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 2, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0.001999964, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0.000999982 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.252 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 2 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.253 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.254 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.255 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.256 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.257 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.258 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.259 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.260 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.260 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:24:36.260 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:24:36.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:24:36.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-02-05T06:24:36.260 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:24:36.347 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.348 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.349 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.350 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.351 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.352 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.353 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.354 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.355 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.356 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:24:36.357 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.358 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.359 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:24:36.360 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:24:36.361 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2024-02-05T06:24:36.361 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:24:36.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:24:36.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:24:36.464 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:24:36.464 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:24:36.465 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:24:36.465 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:24:36.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:24:36.466 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:24:36.466 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:24:36.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:24:36.467 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:24:36.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:24:36.468 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:24:36.468 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:24:36.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:24:36.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:24:36.519 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:24:36.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:36.520 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:36.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:24:36.520 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:24:36.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:24:36.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2024-02-05T06:24:36.521 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2024-02-05T06:24:36.521 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T06:24:36.521 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:24:36.523 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:24:36.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:24:36.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:24:36.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:24:36.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:24:36.524 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:24:36.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:24:36.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:24:36.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:24:36.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:24:36.525 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:24:36.525 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:24:36.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:24:36.526 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:24:36.526 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:24:36.527 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:24:36.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:24:36.528 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:24:36.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:24:36.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:36.529 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:36.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:24:36.529 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:24:36.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:24:36.530 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T06:24:36.531 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:24:36.531 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:36.531 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:36.531 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.548953 2024-02-05T06:24:36.532 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:24:36.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:24:36.532 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:24:36.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T06:24:36.533 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2024-02-05T06:24:36.533 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2024-02-05T06:24:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2024-02-05T06:24:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2024-02-05T06:24:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:527: TEST_dump_scrub_schedule: local poolname=test 2024-02-05T06:24:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:528: TEST_dump_scrub_schedule: local OSDS=3 2024-02-05T06:24:36.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:529: TEST_dump_scrub_schedule: local objects=15 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: TESTDATA=testdata.548953 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-02-05T06:24:36.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2024-02-05T06:24:36.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:24:36.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:24:36.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:24:36.908 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:24:36.908 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:36.908 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:36.908 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:36.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-02-05T06:24:37.083 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:24:37.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:24:37.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:24:37.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:24:37.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:24:37.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:24:37.085 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-mon.a.asok 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:24:37.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.548953/ceph-mon.a.asok config get fsid 2024-02-05T06:24:37.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:24:37.165 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-mon.a.asok 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:24:37.166 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.548953/ceph-mon.a.asok config get mon_host 2024-02-05T06:24:37.244 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:534: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x 2024-02-05T06:24:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-02-05T06:24:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:24:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:24:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:24:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-02-05T06:24:37.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:24:38.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:24:38.053 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:24:38.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:545: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:24:38.079 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: expr 3 - 1 2024-02-05T06:24:38.080 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: seq 0 2 2024-02-05T06:24:38.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:24:38.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:24:38.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:24:38.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:24:38.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:24:38.084 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:24:38.085 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:24:38.086 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:24:38.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:24:38.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:24:38.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-02-05T06:24:38.087 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:24:38.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c24d897d-1029-4b72-bdd5-ab39cb498ecc 2024-02-05T06:24:38.087 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 c24d897d-1029-4b72-bdd5-ab39cb498ecc 2024-02-05T06:24:38.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c24d897d-1029-4b72-bdd5-ab39cb498ecc' 2024-02-05T06:24:38.088 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:24:38.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmf8BlZRmUBhAAerAbTIzV8/eHLHiNwPoNpQ== 2024-02-05T06:24:38.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmf8BlZRmUBhAAerAbTIzV8/eHLHiNwPoNpQ=="}' 2024-02-05T06:24:38.100 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c24d897d-1029-4b72-bdd5-ab39cb498ecc -i td/osd-scrub-test/0/new.json 2024-02-05T06:24:38.300 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:24:38.310 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-02-05T06:24:38.311 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQAmf8BlZRmUBhAAerAbTIzV8/eHLHiNwPoNpQ== --osd-uuid c24d897d-1029-4b72-bdd5-ab39cb498ecc 2024-02-05T06:24:38.341 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:38.351+0000 7feda7e8a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:38.351 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:38.361+0000 7feda7e8a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:38.352 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:38.362+0000 7feda7e8a4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:38.353 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:38.363+0000 7feda7e8a4c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:24:38.353 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:38.363+0000 7feda7e8a4c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:24:38.353 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:38.363+0000 7feda7e8a4c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:24:38.435 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:38.444+0000 7feda7e8a4c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T06:24:44.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-02-05T06:24:44.005 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:24:44.006 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:24:44.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:24:44.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:24:44.347 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:24:44.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:24:44.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:24:44.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:24:44.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:24:44.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:24:44.375 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:44.385+0000 7fbbfcdad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:44.377 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:44.387+0000 7fbbfcdad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:44.378 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:44.388+0000 7fbbfcdad4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:44.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:24:44.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:24:44.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:24:44.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:24:44.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:24:44.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:44.612 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:24:44.613 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:24:44.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:44.613 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:44.877 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:44.901 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:44.911+0000 7fbbfcdad4c0 -1 Falling back to public interface 2024-02-05T06:24:45.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:45.878 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:45.878 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:24:45.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:24:45.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:45.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:46.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:47.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:47.149 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:47.149 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:24:47.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:24:47.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:47.150 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:47.276 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:47.286+0000 7fbbfcdad4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:24:47.460 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:48.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:48.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:48.462 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:24:48.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:24:48.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:48.463 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:24:48.742 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2952047945,v1:127.0.0.1:6803/2952047945] [v2:127.0.0.1:6804/2952047945,v1:127.0.0.1:6805/2952047945] exists,up c24d897d-1029-4b72-bdd5-ab39cb498ecc 2024-02-05T06:24:48.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:24:48.746 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-02-05T06:24:48.747 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:24:48.748 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bfb50e9e-7f09-4453-ad00-c2d7a085cf1a 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bfb50e9e-7f09-4453-ad00-c2d7a085cf1a' 2024-02-05T06:24:48.749 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 bfb50e9e-7f09-4453-ad00-c2d7a085cf1a 2024-02-05T06:24:48.750 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:24:48.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwf8BlWr8ALhAAh1IS8rl6WujY095xetonLQ== 2024-02-05T06:24:48.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwf8BlWr8ALhAAh1IS8rl6WujY095xetonLQ=="}' 2024-02-05T06:24:48.762 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bfb50e9e-7f09-4453-ad00-c2d7a085cf1a -i td/osd-scrub-test/1/new.json 2024-02-05T06:24:49.076 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:24:49.087 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-02-05T06:24:49.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQAwf8BlWr8ALhAAh1IS8rl6WujY095xetonLQ== --osd-uuid bfb50e9e-7f09-4453-ad00-c2d7a085cf1a 2024-02-05T06:24:49.118 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:49.128+0000 7f7198cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:49.152 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:49.162+0000 7f7198cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:49.153 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:49.163+0000 7f7198cb64c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:49.154 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:49.163+0000 7f7198cb64c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:24:49.154 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:49.163+0000 7f7198cb64c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:24:49.154 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:49.163+0000 7f7198cb64c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:24:49.227 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:49.237+0000 7f7198cb64c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T06:24:55.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-02-05T06:24:55.056 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:24:55.057 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:24:55.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:24:55.058 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:24:55.403 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:24:55.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:24:55.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:24:55.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:24:55.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:24:55.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:24:55.432 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:55.441+0000 7fb91eba94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:55.434 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:55.444+0000 7fb91eba94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:55.435 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:55.445+0000 7fb91eba94c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:24:55.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:24:55.671 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:24:55.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:24:55.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:24:55.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:24:55.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:55.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:24:55.672 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:24:55.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:55.673 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:24:55.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:55.956 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:55.966+0000 7fb91eba94c0 -1 Falling back to public interface 2024-02-05T06:24:56.940 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:56.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:56.941 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:24:56.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:24:56.941 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:56.942 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:24:57.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:58.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:58.218 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:58.219 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:24:58.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:24:58.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:58.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:24:58.445 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:24:58.454+0000 7fb91eba94c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:24:58.492 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:24:59.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:24:59.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:24:59.494 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:24:59.494 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:24:59.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:24:59.495 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:24:59.793 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2148368637,v1:127.0.0.1:6811/2148368637] [v2:127.0.0.1:6812/2148368637,v1:127.0.0.1:6813/2148368637] exists,up bfb50e9e-7f09-4453-ad00-c2d7a085cf1a 2024-02-05T06:24:59.797 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:547: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:549: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/2 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:24:59.798 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:24:59.799 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2' 2024-02-05T06:24:59.800 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/2 2024-02-05T06:24:59.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:24:59.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8aeabe2b-e339-42e1-88b1-f6d3a3c75f65 2024-02-05T06:24:59.801 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 8aeabe2b-e339-42e1-88b1-f6d3a3c75f65 2024-02-05T06:24:59.801 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8aeabe2b-e339-42e1-88b1-f6d3a3c75f65' 2024-02-05T06:24:59.801 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:24:59.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7f8BlMVoEMRAA4tuVcRnZ3eI8ef0K8gLA5Q== 2024-02-05T06:24:59.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7f8BlMVoEMRAA4tuVcRnZ3eI8ef0K8gLA5Q=="}' 2024-02-05T06:24:59.813 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8aeabe2b-e339-42e1-88b1-f6d3a3c75f65 -i td/osd-scrub-test/2/new.json 2024-02-05T06:25:00.140 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:25:00.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/2/new.json 2024-02-05T06:25:00.152 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 --mkfs --key AQA7f8BlMVoEMRAA4tuVcRnZ3eI8ef0K8gLA5Q== --osd-uuid 8aeabe2b-e339-42e1-88b1-f6d3a3c75f65 2024-02-05T06:25:00.183 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:00.192+0000 7fc01d01d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:00.217 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:00.226+0000 7fc01d01d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:00.218 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:00.227+0000 7fc01d01d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:00.218 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:00.228+0000 7fc01d01d4c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:25:00.219 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:00.228+0000 7fc01d01d4c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:25:00.219 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:00.228+0000 7fc01d01d4c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:25:00.292 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:00.301+0000 7fc01d01d4c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-02-05T06:25:06.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2024-02-05T06:25:06.269 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:25:06.270 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T06:25:06.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-02-05T06:25:06.270 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:25:06.614 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T06:25:06.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-02-05T06:25:06.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_scrub_sleep=0.2 2024-02-05T06:25:06.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:25:06.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:25:06.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:25:06.643 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:06.652+0000 7f73197fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:06.645 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:06.654+0000 7f73197fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:06.646 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:06.655+0000 7f73197fb4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:06.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-02-05T06:25:06.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:25:06.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-02-05T06:25:06.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:25:06.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:25:06.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:06.879 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:25:06.880 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:25:06.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:06.880 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:25:07.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:07.178 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:07.187+0000 7f73197fb4c0 -1 Falling back to public interface 2024-02-05T06:25:08.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:08.143 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:08.144 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:25:08.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:25:08.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:08.144 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:25:08.444 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:09.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:09.446 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:09.446 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:25:09.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:25:09.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:09.447 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:25:09.500 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:09.509+0000 7f73197fb4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T06:25:09.716 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:10.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:10.720 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:10.721 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:25:10.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:25:10.721 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:10.722 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-02-05T06:25:10.998 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/386221305,v1:127.0.0.1:6819/386221305] [v2:127.0.0.1:6820/386221305,v1:127.0.0.1:6821/386221305] exists,up 8aeabe2b-e339-42e1-88b1-f6d3a3c75f65 2024-02-05T06:25:10.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:25:10.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:25:10.999 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:25:11.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:553: TEST_dump_scrub_schedule: create_pool test 1 1 2024-02-05T06:25:11.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:25:11.363 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T06:25:11.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:25:12.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: wait_for_clean 2024-02-05T06:25:12.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:25:12.377 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:25:12.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:25:12.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:25:12.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:25:12.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:25:12.462 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=500 2024-02-05T06:25:12.462 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:25:12.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:25:12.807 INFO:tasks.workunit.client.0.smithi188.stderr:1 2024-02-05T06:25:12.807 INFO:tasks.workunit.client.0.smithi188.stderr:2' 2024-02-05T06:25:12.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:25:12.807 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:25:12.807 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:25:12.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-02-05T06:25:12.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-02-05T06:25:12.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-02-05T06:25:12.932 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:25:12.932 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:25:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-02-05T06:25:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-02-05T06:25:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964' 2024-02-05T06:25:13.032 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:25:13.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-02-05T06:25:13.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-02-05T06:25:13.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-02-05T06:25:13.132 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964 2-64424509442' 2024-02-05T06:25:13.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:25:13.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-02-05T06:25:13.133 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:25:13.133 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:25:13.134 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-02-05T06:25:13.134 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:25:13.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-02-05T06:25:13.135 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-02-05T06:25:13.135 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836486 2024-02-05T06:25:13.135 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:25:13.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-02-05T06:25:13.397 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:25:14.398 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 500 -eq 0 ']' 2024-02-05T06:25:14.398 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:25:14.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836486 2024-02-05T06:25:14.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:25:14.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-02-05T06:25:14.669 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:25:14.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:25:14.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-02-05T06:25:14.670 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:25:14.671 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672964 2024-02-05T06:25:14.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-02-05T06:25:14.672 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-02-05T06:25:14.672 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:25:14.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-02-05T06:25:14.967 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:25:14.968 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-02-05T06:25:14.968 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:25:14.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-02-05T06:25:14.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-02-05T06:25:14.969 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:25:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-02-05T06:25:14.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-02-05T06:25:14.971 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509442 2024-02-05T06:25:14.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-02-05T06:25:15.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-02-05T06:25:15.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:25:15.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:25:15.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:25:15.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:25:15.563 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:25:15.563 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:25:15.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:25:15.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:25:15.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:25:15.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:25:15.564 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:25:15.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-02-05T06:25:15.822 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:25:15.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:25:15.823 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:25:16.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-02-05T06:25:16.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-02-05T06:25:16.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-02-05T06:25:16.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-02-05T06:25:16.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-02-05T06:25:16.151 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-02-05T06:25:16.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-02-05T06:25:16.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:25:16.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:25:16.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:25:16.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:25:16.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:25:16.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:25:16.253 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:25:16.510 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:25:16.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:25:16.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:25:16.511 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:25:16.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:25:16.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:25:16.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:25:16.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: ceph osd dump 2024-02-05T06:25:16.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2024-02-05T06:25:16.836 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: awk '{ print $2 }' 2024-02-05T06:25:17.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:555: TEST_dump_scrub_schedule: poolid=1 2024-02-05T06:25:17.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:557: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.548953 bs=1032 count=1 2024-02-05T06:25:17.098 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:25:17.098 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:25:17.098 INFO:tasks.workunit.client.0.smithi188.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.4679e-05 s, 23.1 MB/s 2024-02-05T06:25:17.099 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: seq 1 15 2024-02-05T06:25:17.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:17.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.548953 2024-02-05T06:25:17.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:17.257 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.548953 2024-02-05T06:25:17.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:17.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.548953 2024-02-05T06:25:17.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:17.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.548953 2024-02-05T06:25:17.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:17.719 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.548953 2024-02-05T06:25:17.852 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:17.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.548953 2024-02-05T06:25:17.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:17.978 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.548953 2024-02-05T06:25:18.113 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:18.114 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.548953 2024-02-05T06:25:18.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:18.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.548953 2024-02-05T06:25:18.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:18.415 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.548953 2024-02-05T06:25:18.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:18.549 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.548953 2024-02-05T06:25:18.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:18.683 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.548953 2024-02-05T06:25:18.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:18.817 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.548953 2024-02-05T06:25:18.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:18.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.548953 2024-02-05T06:25:19.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:558: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2024-02-05T06:25:19.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.548953 2024-02-05T06:25:19.211 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: rm -f testdata.548953 2024-02-05T06:25:19.212 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: local pgid=1.0 2024-02-05T06:25:19.212 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: date -Ins 2024-02-05T06:25:19.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: local now_is=2024-02-05T06:25:19,223081236+00:00 2024-02-05T06:25:19.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2024-02-05T06:25:19.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: declare -A expct_starting 2024-02-05T06:25:19.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:574: TEST_dump_scrub_schedule: declare -A sched_data 2024-02-05T06:25:19.214 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:575: TEST_dump_scrub_schedule: extract_published_sch 1.0 2024-02-05T06:25:19,223081236+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2024-02-05T06:25:19.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2024-02-05T06:25:19.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-02-05T06:25:19.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-02-05T06:25:19,223081236+00:00 2024-02-05T06:25:19.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2024-02-05T06:25:19.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-02-05T06:25:19.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-02-05T06:25:19.215 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-02-05T06:25:19.462 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:25:19.580 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:25:19.580 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=19 2024-02-05T06:25:19.580 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=28 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:25:11.317' 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:25:11.317588+0000' 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=null 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:25:11.317588+0000' 2024-02-05T06:25:19.581 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:25:19.582 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:25:19.582 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=19 2024-02-05T06:25:19.582 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=12 2024-02-05T06:25:19.582 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:25:19.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:576: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2024-02-05T06:25:19.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:170: schedule_against_expected: local -n dict=sched_data 2024-02-05T06:25:19.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:171: schedule_against_expected: local -n ep=expct_starting 2024-02-05T06:25:19.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:172: schedule_against_expected: local extr_dbg=1 2024-02-05T06:25:19.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:175: schedule_against_expected: local saved_echo_flag=x 2024-02-05T06:25:19.585 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:176: schedule_against_expected: set +x 2024-02-05T06:25:19.585 INFO:tasks.workunit.client.0.smithi188.stdout:-- - comparing: 2024-02-05T06:25:19.585 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2024-02-05T06:25:19.585 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_is_future expected: true in actual: true 2024-02-05T06:25:19.586 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_active expected: false in actual: false 2024-02-05T06:25:19.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:195: schedule_against_expected: return 0 2024-02-05T06:25:19.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:577: TEST_dump_scrub_schedule: (( 0 == 0 )) 2024-02-05T06:25:19.586 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:578: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2024-02-05T06:25:19.586 INFO:tasks.workunit.client.0.smithi188.stdout:last-scrub --- 0x0 2024-02-05T06:25:19.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:584: TEST_dump_scrub_schedule: saved_last_stamp=2024-02-05T06:25:11.317588+0000 2024-02-05T06:25:19.587 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:585: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2024-02-05T06:25:19.677 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:25:19.677 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:25:19.677 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:19.691 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:25:19.692 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:25:19.692 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:19.705 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2: { 2024-02-05T06:25:19.705 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:25:19.705 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:19.715 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:586: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2024-02-05T06:25:19.828 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:25:19.828 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:25:19.828 INFO:tasks.workunit.client.0.smithi188.stdout: "must": true, 2024-02-05T06:25:19.829 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "0.000000" 2024-02-05T06:25:19.829 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:19.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:591: TEST_dump_scrub_schedule: sleep 5 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:592: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-02-05T06:25:11.317588+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T06:25:11.317588+0000 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-02-05T06:25:24.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:25:24.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:25:24.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:25:24.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:25:24.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:25:24.841 INFO:tasks.workunit.client.0.smithi188.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2024-02-05T06:25:11.317588+0000 (10 retries) 2024-02-05T06:25:25.599 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:25:25.717 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=19 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=39 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:25:20.323' 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=1 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:25:20.323175+0000' 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='19x15' 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=null 2024-02-05T06:25:25.718 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='queued for deep scrub' 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='0' 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=false 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=false 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=19 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=29 2024-02-05T06:25:25.719 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:25:25.721 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 1 ~ false / 39 / 29 / true / 2024-02-05T06:25:20.323175+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:25:25.721 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2024-02-05T06:25:25.721 INFO:tasks.workunit.client.0.smithi188.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:596: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:597: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:598: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-02-05T06:25:11.317588+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T06:25:11.317588+0000 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_dmp_duration 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:25:25.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:25:25.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:25:25.724 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:25:25.724 INFO:tasks.workunit.client.0.smithi188.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2024-02-05T06:25:11.317588+0000 (10 retries) 2024-02-05T06:25:26.476 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:25:26.592 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=19 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=39 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:25:20.323' 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=1 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:25:20.323175+0000' 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='19x15' 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=null 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:25:26.593 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=1 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:25:20.323175+0000' 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=19 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=38 2024-02-05T06:25:26.594 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:25:26.596 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 1 ~ false / 39 / 38 / true / 2024-02-05T06:25:20.323175+0000 / scrub scheduled %%% dmp_last_duration dmp_last_duration_neg 2024-02-05T06:25:26.596 INFO:tasks.workunit.client.0.smithi188.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2024-02-05T06:25:26.596 INFO:tasks.workunit.client.0.smithi188.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2024-02-05T06:25:26.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:25:26.597 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: sleep 2 2024-02-05T06:25:28.598 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:607: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_chunk_max 3 2024-02-05T06:25:28.700 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:25:28.700 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-02-05T06:25:28.700 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:28.714 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:25:28.714 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-02-05T06:25:28.715 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:28.728 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2: { 2024-02-05T06:25:28.728 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_chunk_max = '' (not observed, change may require restart) " 2024-02-05T06:25:28.729 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:28.737 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:608: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 1.0 2024-02-05T06:25:28.851 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0: { 2024-02-05T06:25:28.851 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:25:28.852 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:28.866 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1: { 2024-02-05T06:25:28.866 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:25:28.866 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:28.879 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2: { 2024-02-05T06:25:28.879 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_sleep = '' " 2024-02-05T06:25:28.879 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:28.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:609: TEST_dump_scrub_schedule: ceph osd set noscrub 2024-02-05T06:25:29.275 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:25:29.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:610: TEST_dump_scrub_schedule: sleep 2 2024-02-05T06:25:31.288 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:611: TEST_dump_scrub_schedule: saved_last_stamp=2024-02-05T06:25:20.323175+0000 2024-02-05T06:25:31.289 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2024-02-05T06:25:31.392 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:25:31.392 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": false, 2024-02-05T06:25:31.392 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:25:31.392 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-29T06:23:51.401154+0000" 2024-02-05T06:25:31.393 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:31.401 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: sleep 1 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:615: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-02-05T06:25:20.323175+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T06:25:20.323175+0000 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_scrub_peri_sched 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:25:32.404 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:25:32.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:25:32.405 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:25:32.405 INFO:tasks.workunit.client.0.smithi188.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2024-02-05T06:25:20.323175+0000 (10 retries) 2024-02-05T06:25:33.165 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=20 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=43 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='queued for scrub' 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='0' 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=1 2024-02-05T06:25:33.283 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-01-29T06:23:51.401154+0000' 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='19x15' 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=false 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=false 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=null 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=1 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:25:20.323175+0000' 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=20 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=40 2024-02-05T06:25:33.284 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:25:33.286 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 1 ~ false / 43 / 40 / false / 2024-01-29T06:23:51.401154+0000 / queued for scrub %%% query_is_future 2024-02-05T06:25:33.286 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2024-02-05T06:25:33.287 INFO:tasks.workunit.client.0.smithi188.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2024-02-05T06:25:33.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:25:33.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:629: TEST_dump_scrub_schedule: saved_last_stamp=2024-01-29T06:23:51.401154+0000 2024-02-05T06:25:33.287 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:630: TEST_dump_scrub_schedule: ceph osd unset noscrub 2024-02-05T06:25:33.657 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is unset 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:632: TEST_dump_scrub_schedule: declare -A cond_active 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:633: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:634: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-01-29T06:23:51.401154+0000 cond_active 'WaitingActive ' sched_data 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-01-29T06:23:51.401154+0000 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:25:33.669 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:25:33.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:25:33.670 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:25:33.670 INFO:tasks.workunit.client.0.smithi188.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-01-29T06:23:51.401154+0000 (10 retries) 2024-02-05T06:25:34.421 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=21 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=47 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=true 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrubbing' 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='0' 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=1 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-01-29T06:23:51.401154+0000' 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='19x15' 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=false 2024-02-05T06:25:34.560 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=false 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=null 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=1 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:25:20.323175+0000' 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=20 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=40 2024-02-05T06:25:34.561 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:25:34.563 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 1 ~ true / 47 / 40 / false / 2024-01-29T06:23:51.401154+0000 / scrubbing %%% query_active 2024-02-05T06:25:34.563 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_active: negation:0 # expected: true # in actual: true 2024-02-05T06:25:34.563 INFO:tasks.workunit.client.0.smithi188.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2024-02-05T06:25:34.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:25:34.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:638: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2024-02-05T06:25:34.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:638: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2024-02-05T06:25:34.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: sched_data=() 2024-02-05T06:25:34.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:640: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2024-01-29T06:23:51.401154+0000 cond_active_dmp 'WaitingActive ' sched_data 2024-02-05T06:25:34.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.0 2024-02-05T06:25:34.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:25:34.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-01-29T06:23:51.401154+0000 2024-02-05T06:25:34.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=cond_active_dmp 2024-02-05T06:25:34.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sched_data 2024-02-05T06:25:34.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:25:34.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:25:34.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:25:34.567 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:25:34.567 INFO:tasks.workunit.client.0.smithi188.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2024-01-29T06:23:51.401154+0000 (10 retries) 2024-02-05T06:25:35.345 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:25:35.463 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:25:35.463 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=21 2024-02-05T06:25:35.463 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=53 2024-02-05T06:25:35.463 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:25:35.440' 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=2 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:25:35.440881+0000' 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='19x15' 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=null 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=1 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:25:20.323175+0000' 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:25:35.464 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:25:35.465 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=20 2024-02-05T06:25:35.465 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=40 2024-02-05T06:25:35.467 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:25:35.468 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 1 ~ false / 53 / 40 / true / 2024-02-05T06:25:35.440881+0000 / scrub scheduled %%% dmp_state_has_scrubbing query_active 2024-02-05T06:25:35.468 INFO:tasks.workunit.client.0.smithi188.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: false 2024-02-05T06:25:35.468 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_active: negation:0 # expected: false # in actual: false 2024-02-05T06:25:35.468 INFO:tasks.workunit.client.0.smithi188.stdout:WaitingActive - 'query_active' actual value (false) matches expected (false) (negation: 0) 2024-02-05T06:25:35.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:149: wait_any_cond: return 0 2024-02-05T06:25:35.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: perf_counters td/osd-scrub-test 3 2024-02-05T06:25:35.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-02-05T06:25:35.469 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2024-02-05T06:25:35.469 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2024-02-05T06:25:35.469 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2024-02-05T06:25:35.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:25:35.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-02-05T06:25:35.470 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.563 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.564 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.565 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:25:35.566 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.567 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.568 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:25:35.569 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.570 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.571 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.572 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.573 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.574 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.575 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-02-05T06:25:35.576 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.665 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.666 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.667 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:25:35.668 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 1, 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 1, 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 1, 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 1, 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0.035999350, 2024-02-05T06:25:35.672 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0.035999350 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 1, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.673 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 1, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 2 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.674 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.675 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.676 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.677 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 1, 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 1, 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 1, 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 1, 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 1.002981880, 2024-02-05T06:25:35.678 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 1.002981880 2024-02-05T06:25:35.679 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.679 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.679 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.679 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.679 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.679 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.679 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.680 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.680 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 1, 2024-02-05T06:25:35.680 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.680 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 1, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 1, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0.000999982, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0.000999982 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.681 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.682 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.682 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.682 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 2 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:25:35.683 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:35.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:25:35.684 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2024-02-05T06:25:35.685 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:25:35.765 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.766 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.767 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.768 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.769 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.770 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.771 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.772 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.773 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.774 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:25:35.775 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:25:35.776 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:25:35.777 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.778 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:25:35.778 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:25:35.778 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:25:35.778 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_dump_scrub_schedule ------------------' 2024-02-05T06:25:35.778 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Teardown Test TEST_dump_scrub_schedule ------------------ 2024-02-05T06:25:35.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-02-05T06:25:35.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:25:35.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:25:35.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:25:35.779 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:25:35.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:25:35.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:25:35.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:25:35.780 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:25:35.882 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:25:35.882 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:25:35.883 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:25:35.883 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:25:35.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:25:35.884 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:25:35.884 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:25:35.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:35.885 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:25:35.886 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:25:35.886 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:35.887 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:25:35.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:25:35.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:25:35.893 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:25:35.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:35.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:35.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:25:35.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:25:35.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:25:35.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_dump_scrub_schedule ------------------' 2024-02-05T06:25:35.895 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Complete Test TEST_dump_scrub_schedule ------------------ 2024-02-05T06:25:35.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T06:25:35.895 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Prepare Test TEST_interval_changes ------------------- 2024-02-05T06:25:35.897 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_interval_changes -------------------' 2024-02-05T06:25:35.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T06:25:35.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T06:25:35.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T06:25:35.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:25:35.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:25:35.898 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:25:35.898 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:25:35.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:25:35.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:25:35.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:25:35.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:25:35.899 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:25:35.899 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:25:35.900 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:25:35.900 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:25:35.901 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:25:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:25:35.902 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:25:35.903 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:25:35.904 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:35.904 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:25:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:25:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:25:35.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T06:25:35.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:25:35.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:35.905 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:35.906 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.548953 2024-02-05T06:25:35.907 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:25:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:25:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:25:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T06:25:35.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_interval_changes -----------------------' 2024-02-05T06:25:35.907 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Run Test TEST_interval_changes ----------------------- 2024-02-05T06:25:35.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_interval_changes td/osd-scrub-test 2024-02-05T06:25:35.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:149: TEST_interval_changes: local poolname=test 2024-02-05T06:25:35.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: local OSDS=2 2024-02-05T06:25:35.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:151: TEST_interval_changes: local objects=10 2024-02-05T06:25:35.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: expr 24 '*' 60 '*' 60 2024-02-05T06:25:35.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:153: TEST_interval_changes: local day=86400 2024-02-05T06:25:35.909 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: expr 86400 '*' 7 2024-02-05T06:25:35.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:154: TEST_interval_changes: local week=604800 2024-02-05T06:25:35.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: local min_interval=86400 2024-02-05T06:25:35.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:156: TEST_interval_changes: local max_interval=604800 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: TESTDATA=testdata.548953 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:162: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-test 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-test/a 2024-02-05T06:25:35.911 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2024-02-05T06:25:36.312 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-02-05T06:25:36.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:25:36.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:25:36.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:25:36.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:36.313 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:36.313 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:25:36.313 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-02-05T06:25:36.488 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-mon.a.asok 2024-02-05T06:25:36.490 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:25:36.491 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.548953/ceph-mon.a.asok config get fsid 2024-02-05T06:25:36.568 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-mon.a.asok 2024-02-05T06:25:36.569 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-02-05T06:25:36.570 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.548953/ceph-mon.a.asok config get mon_host 2024-02-05T06:25:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:163: TEST_interval_changes: run_mgr td/osd-scrub-test x 2024-02-05T06:25:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-test 2024-02-05T06:25:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-02-05T06:25:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-02-05T06:25:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-02-05T06:25:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-test/x 2024-02-05T06:25:36.648 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-02-05T06:25:37.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-02-05T06:25:37.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:25:37.449 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:25:37.450 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:25:37.450 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:37.450 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:37.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:25:37.450 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:25:37.450 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-02-05T06:25:37.476 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: expr 2 - 1 2024-02-05T06:25:37.476 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: seq 0 1 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/0 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2024-02-05T06:25:37.481 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:25:37.482 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/0 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=710fcb26-5b20-4cc1-b76f-a7c12e0444be 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 710fcb26-5b20-4cc1-b76f-a7c12e0444be' 2024-02-05T06:25:37.483 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 710fcb26-5b20-4cc1-b76f-a7c12e0444be 2024-02-05T06:25:37.484 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:25:37.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhf8BlNhweHhAAa60fY5Ml4aSqYsehInbPBw== 2024-02-05T06:25:37.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhf8BlNhweHhAAa60fY5Ml4aSqYsehInbPBw=="}' 2024-02-05T06:25:37.497 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 710fcb26-5b20-4cc1-b76f-a7c12e0444be -i td/osd-scrub-test/0/new.json 2024-02-05T06:25:37.696 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:25:37.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/0/new.json 2024-02-05T06:25:37.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQBhf8BlNhweHhAAa60fY5Ml4aSqYsehInbPBw== --osd-uuid 710fcb26-5b20-4cc1-b76f-a7c12e0444be 2024-02-05T06:25:37.735 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:37.743+0000 7f43eae1d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:37.747 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:37.755+0000 7f43eae1d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:37.748 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:37.757+0000 7f43eae1d4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:37.749 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:37.757+0000 7f43eae1d4c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:25:37.749 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:37.757+0000 7f43eae1d4c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:25:37.749 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:37.757+0000 7f43eae1d4c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:25:37.839 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:37.847+0000 7f43eae1d4c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T06:25:43.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2024-02-05T06:25:43.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:25:43.333 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-02-05T06:25:43.334 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:25:43.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:25:43.679 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:25:43.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-02-05T06:25:43.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:25:43.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:25:43.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:25:43.680 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:25:43.704 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:43.712+0000 7fadafada4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:43.706 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:43.714+0000 7fadafada4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:43.707 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:43.715+0000 7fadafada4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:25:43.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-02-05T06:25:43.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:25:43.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-02-05T06:25:43.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:25:43.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:25:43.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:43.963 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:25:43.964 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:25:43.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:43.964 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:44.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:44.251 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:44.259+0000 7fadafada4c0 -1 Falling back to public interface 2024-02-05T06:25:45.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:45.231 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:45.231 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:25:45.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:25:45.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:45.232 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:45.503 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:46.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:46.504 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:46.505 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:25:46.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:25:46.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:46.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:46.664 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:25:46.672+0000 7fadafada4c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:25:46.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:47.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:47.841 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:47.841 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:25:47.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:25:47.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:47.842 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:48.116 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:49.117 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:49.118 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:25:49.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:49.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:25:49.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:49.118 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:49.522 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:50.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:50.523 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:50.523 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:25:50.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:25:50.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:50.524 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:50.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:51.794 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:51.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:51.795 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:25:51.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:25:51.795 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:51.796 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:52.070 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:53.071 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:53.071 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:25:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:25:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:53.072 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:53.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:54.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:54.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:54.350 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:25:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:25:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:54.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:54.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:55.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:55.707 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:55.707 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:25:55.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:25:55.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:55.708 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:55.980 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:56.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:56.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:56.984 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:25:56.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:25:56.984 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:56.985 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:57.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:58.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:58.292 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:58.292 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:25:58.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:25:58.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:58.293 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:58.592 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:25:59.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:25:59.593 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:25:59.594 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:25:59.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:25:59.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:25:59.594 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:25:59.888 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:00.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:00.889 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:00.889 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:26:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:26:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:00.890 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:26:01.332 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:02.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:02.334 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:02.334 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:26:02.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:26:02.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:02.335 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:26:03.077 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:04.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:04.078 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:04.079 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:26:04.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:26:04.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:04.079 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:26:04.664 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:05.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:05.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:05.666 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:26:05.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:26:05.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:05.666 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:26:06.641 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:07.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:07.642 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:07.642 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:26:07.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:26:07.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:07.643 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:26:08.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:09.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:09.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:09.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:26:09.236 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:26:09.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:09.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:26:09.615 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:10.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:10.616 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:10.617 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:26:10.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:26:10.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:10.618 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-02-05T06:26:10.888 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/580481997,v1:127.0.0.1:6803/580481997] [v2:127.0.0.1:6804/580481997,v1:127.0.0.1:6805/580481997] exists,up 710fcb26-5b20-4cc1-b76f-a7c12e0444be 2024-02-05T06:26:10.892 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:166: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-test 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-test/1 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-02-05T06:26:10.893 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' 2024-02-05T06:26:10.894 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-test/1 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-02-05T06:26:10.895 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ce5f056-a5cc-49c2-aae5-ac931b97b937 2024-02-05T06:26:10.896 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 6ce5f056-a5cc-49c2-aae5-ac931b97b937 2024-02-05T06:26:10.896 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6ce5f056-a5cc-49c2-aae5-ac931b97b937' 2024-02-05T06:26:10.896 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-02-05T06:26:10.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCf8BlQZmPNhAAoWgw62Ca+kHLfg04F2647A== 2024-02-05T06:26:10.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCf8BlQZmPNhAAoWgw62Ca+kHLfg04F2647A=="}' 2024-02-05T06:26:10.908 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ce5f056-a5cc-49c2-aae5-ac931b97b937 -i td/osd-scrub-test/1/new.json 2024-02-05T06:26:11.223 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:26:11.234 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-test/1/new.json 2024-02-05T06:26:11.235 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQCCf8BlQZmPNhAAoWgw62Ca+kHLfg04F2647A== --osd-uuid 6ce5f056-a5cc-49c2-aae5-ac931b97b937 2024-02-05T06:26:11.263 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:11.270+0000 7f405de0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:11.299 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:11.307+0000 7f405de0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:11.300 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:11.308+0000 7f405de0f4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:11.308+0000 7f405de0f4c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:26:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:11.308+0000 7f405de0f4c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:26:11.301 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:11.308+0000 7f405de0f4c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:26:11.374 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:11.381+0000 7f405de0f4c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T06:26:17.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2024-02-05T06:26:17.185 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-02-05T06:26:17.186 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:26:17.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-02-05T06:26:17.186 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-02-05T06:26:17.533 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:26:17.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-02-05T06:26:17.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=43281632-48a0-49b4-9a38-5e183b7b3ca3 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.548953/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2024-02-05T06:26:17.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-02-05T06:26:17.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-02-05T06:26:17.534 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-02-05T06:26:17.559 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:17.566+0000 7f42ec80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:17.561 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:17.569+0000 7f42ec80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:17.562 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:17.570+0000 7f42ec80e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:26:17.802 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-02-05T06:26:17.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-02-05T06:26:17.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-02-05T06:26:17.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-02-05T06:26:17.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-02-05T06:26:17.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:17.803 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-02-05T06:26:17.803 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:26:17.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:17.804 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:18.074 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:18.089 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:18.097+0000 7f42ec80e4c0 -1 Falling back to public interface 2024-02-05T06:26:19.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:19.075 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:19.075 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:26:19.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-02-05T06:26:19.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:19.076 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:19.345 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:20.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:20.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:20.347 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:26:20.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-02-05T06:26:20.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:20.348 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:20.415 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:26:20.422+0000 7f42ec80e4c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:26:20.623 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:21.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:21.625 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:21.626 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:26:21.626 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-02-05T06:26:21.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:21.627 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:21.903 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:22.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:22.904 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:22.904 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:26:22.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-02-05T06:26:22.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:22.905 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:23.219 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:24.220 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:24.221 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:24.221 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:26:24.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-02-05T06:26:24.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:24.222 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:24.498 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:25.499 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:25.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:25.500 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:26:25.500 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-02-05T06:26:25.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:25.501 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:25.837 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:26.838 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:26.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:26.839 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:26:26.839 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-02-05T06:26:26.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:26.840 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:27.226 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:28.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:28.228 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:28.228 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:26:28.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-02-05T06:26:28.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:28.229 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:28.505 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:29.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:29.506 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:29.506 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:26:29.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-02-05T06:26:29.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:29.507 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:29.820 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:30.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:30.822 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:30.822 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:26:30.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-02-05T06:26:30.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:30.823 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:31.096 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:32.097 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:32.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:32.098 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:26:32.098 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-02-05T06:26:32.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:32.099 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:32.408 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:33.409 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:33.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:33.410 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:26:33.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-02-05T06:26:33.410 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:33.411 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:33.828 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:34.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:34.829 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:34.830 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:26:34.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-02-05T06:26:34.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:34.830 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:35.581 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:36.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:36.583 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:36.583 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:26:36.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-02-05T06:26:36.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:36.584 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:37.325 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:38.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:38.327 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:38.327 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:26:38.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-02-05T06:26:38.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:38.328 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:38.947 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:39.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:39.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:39.949 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:26:39.949 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-02-05T06:26:39.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:39.950 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:40.907 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:41.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:41.909 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:41.909 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:26:41.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-02-05T06:26:41.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:41.910 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:42.249 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:43.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:43.251 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:43.252 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:26:43.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-02-05T06:26:43.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:43.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:43.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-02-05T06:26:44.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-02-05T06:26:44.705 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 500 )) 2024-02-05T06:26:44.706 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:26:44.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-02-05T06:26:44.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-02-05T06:26:44.706 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-02-05T06:26:44.990 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1519424055,v1:127.0.0.1:6811/1519424055] [v2:127.0.0.1:6812/1519424055,v1:127.0.0.1:6813/1519424055] exists,up 6ce5f056-a5cc-49c2-aae5-ac931b97b937 2024-02-05T06:26:44.990 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-02-05T06:26:44.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-02-05T06:26:44.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-02-05T06:26:44.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: create_pool test 1 1 2024-02-05T06:26:44.991 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-02-05T06:26:45.365 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T06:26:45.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-02-05T06:26:46.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:171: TEST_interval_changes: wait_for_clean 2024-02-05T06:26:46.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-02-05T06:26:46.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-02-05T06:26:46.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-02-05T06:26:46.378 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-02-05T06:26:46.379 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-02-05T06:26:46.379 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-02-05T06:26:46.379 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-02-05T06:26:46.379 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-02-05T06:26:46.379 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-02-05T06:26:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-02-05T06:26:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-02-05T06:26:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-02-05T06:26:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-02-05T06:26:46.471 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=500 2024-02-05T06:26:46.471 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-02-05T06:26:46.768 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-02-05T06:26:46.768 INFO:tasks.workunit.client.0.smithi188.stderr:1' 2024-02-05T06:26:46.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-02-05T06:26:46.769 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:26:46.769 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-02-05T06:26:46.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836495 2024-02-05T06:26:46.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836495 2024-02-05T06:26:46.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495' 2024-02-05T06:26:46.868 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-02-05T06:26:46.868 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-02-05T06:26:46.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672962 2024-02-05T06:26:46.969 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672962 2024-02-05T06:26:46.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836495 1-42949672962' 2024-02-05T06:26:46.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:26:46.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836495 2024-02-05T06:26:46.970 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:26:46.970 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-02-05T06:26:46.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836495 2024-02-05T06:26:46.971 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:26:46.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836495 2024-02-05T06:26:46.972 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-02-05T06:26:46.972 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836495 2024-02-05T06:26:46.973 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:26:47.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836495 2024-02-05T06:26:47.237 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-02-05T06:26:48.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 500 -eq 0 ']' 2024-02-05T06:26:48.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-02-05T06:26:48.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836495 2024-02-05T06:26:48.511 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-02-05T06:26:48.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672962 2024-02-05T06:26:48.512 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-02-05T06:26:48.512 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-02-05T06:26:48.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672962 2024-02-05T06:26:48.513 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-02-05T06:26:48.514 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672962 2024-02-05T06:26:48.514 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672962 2024-02-05T06:26:48.515 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2024-02-05T06:26:48.515 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-02-05T06:26:48.779 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672962 -lt 42949672962 2024-02-05T06:26:48.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-02-05T06:26:48.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:26:48.780 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-02-05T06:26:49.110 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-02-05T06:26:49.412 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-02-05T06:26:49.412 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-02-05T06:26:49.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-02-05T06:26:49.413 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-02-05T06:26:49.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-02-05T06:26:49.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-02-05T06:26:49.741 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-02-05T06:26:49.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: ceph osd dump 2024-02-05T06:26:49.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2024-02-05T06:26:49.742 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: awk '{ print $2 }' 2024-02-05T06:26:50.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: local poolid=1 2024-02-05T06:26:50.006 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: dd if=/dev/urandom of=testdata.548953 bs=1032 count=1 2024-02-05T06:26:50.007 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:26:50.007 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:26:50.007 INFO:tasks.workunit.client.0.smithi188.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.4284e-05 s, 23.3 MB/s 2024-02-05T06:26:50.007 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: seq 1 10 2024-02-05T06:26:50.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.008 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj1 testdata.548953 2024-02-05T06:26:50.147 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.148 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj2 testdata.548953 2024-02-05T06:26:50.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.299 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj3 testdata.548953 2024-02-05T06:26:50.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj4 testdata.548953 2024-02-05T06:26:50.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.542 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj5 testdata.548953 2024-02-05T06:26:50.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.668 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj6 testdata.548953 2024-02-05T06:26:50.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.793 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj7 testdata.548953 2024-02-05T06:26:50.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:50.919 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj8 testdata.548953 2024-02-05T06:26:51.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:51.044 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj9 testdata.548953 2024-02-05T06:26:51.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:175: TEST_interval_changes: for i in `seq 1 $objects` 2024-02-05T06:26:51.170 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: rados -p test put obj10 testdata.548953 2024-02-05T06:26:51.295 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:179: TEST_interval_changes: rm -f testdata.548953 2024-02-05T06:26:51.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: get_primary test obj1 2024-02-05T06:26:51.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-02-05T06:26:51.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-02-05T06:26:51.297 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-02-05T06:26:51.298 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-02-05T06:26:51.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: local primary=1 2024-02-05T06:26:51.565 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:184: TEST_interval_changes: check_dump_scrubs 1 '1 day' '1 week' 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=1 day' 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.1.asok 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T06:26:51.566 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.548953/ceph-osd.1.asok dump_scrubs 2024-02-05T06:26:51.651 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T06:26:51.651 INFO:tasks.workunit.client.0.smithi188.stderr: { 2024-02-05T06:26:51.651 INFO:tasks.workunit.client.0.smithi188.stderr: "pgid": "1.0", 2024-02-05T06:26:51.651 INFO:tasks.workunit.client.0.smithi188.stderr: "sched_time": "2024-02-06T06:26:45.309832+0000", 2024-02-05T06:26:51.651 INFO:tasks.workunit.client.0.smithi188.stderr: "deadline": "2024-02-12T06:26:45.309832+0000", 2024-02-05T06:26:51.651 INFO:tasks.workunit.client.0.smithi188.stderr: "forced": false 2024-02-05T06:26:51.652 INFO:tasks.workunit.client.0.smithi188.stderr: } 2024-02-05T06:26:51.652 INFO:tasks.workunit.client.0.smithi188.stderr:]' 2024-02-05T06:26:51.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-06T06:26:45.309832+0000",' '"deadline":' '"2024-02-12T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:26:51.652 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T06:26:51.654 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-06T06:26:45.309832+0000"' 2024-02-05T06:26:51.654 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-06T06:26:45.309832+0000 2024-02-05T06:26:51.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-06T06:26:45.309832+0000 2024-02-05T06:26:51.655 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:26:51.656 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2024-02-05T06:26:51.657 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-06 = 2024-02-06 2024-02-05T06:26:51.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-06T06:26:45.309832+0000",' '"deadline":' '"2024-02-12T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:26:51.658 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T06:26:51.660 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-12T06:26:45.309832+0000"' 2024-02-05T06:26:51.660 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-12T06:26:45.309832+0000 2024-02-05T06:26:51.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-12T06:26:45.309832+0000 2024-02-05T06:26:51.661 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:26:51.662 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-02-05T06:26:51.663 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-12 = 2024-02-12 2024-02-05T06:26:51.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: get_asok_path osd.1 2024-02-05T06:26:51.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:26:51.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:26:51.664 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:26:51.664 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:26:51.664 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:26:51.664 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.1.asok 2024-02-05T06:26:51.665 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: expr 86400 '*' 2 2024-02-05T06:26:51.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: CEPH_ARGS= 2024-02-05T06:26:51.665 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.548953/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2024-02-05T06:26:51.744 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:26:51.744 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_min_interval = '' " 2024-02-05T06:26:51.744 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:26:51.751 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:188: TEST_interval_changes: sleep 15 2024-02-05T06:27:06.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:189: TEST_interval_changes: check_dump_scrubs 1 '2 days' '1 week' 2024-02-05T06:27:06.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T06:27:06.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-02-05T06:27:06.755 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=1 week' 2024-02-05T06:27:06.755 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.1.asok 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T06:27:06.756 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.548953/ceph-osd.1.asok dump_scrubs 2024-02-05T06:27:06.849 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T06:27:06.850 INFO:tasks.workunit.client.0.smithi188.stderr: { 2024-02-05T06:27:06.850 INFO:tasks.workunit.client.0.smithi188.stderr: "pgid": "1.0", 2024-02-05T06:27:06.850 INFO:tasks.workunit.client.0.smithi188.stderr: "sched_time": "2024-02-07T06:26:45.309832+0000", 2024-02-05T06:27:06.850 INFO:tasks.workunit.client.0.smithi188.stderr: "deadline": "2024-02-12T06:26:45.309832+0000", 2024-02-05T06:27:06.850 INFO:tasks.workunit.client.0.smithi188.stderr: "forced": false 2024-02-05T06:27:06.850 INFO:tasks.workunit.client.0.smithi188.stderr: } 2024-02-05T06:27:06.850 INFO:tasks.workunit.client.0.smithi188.stderr:]' 2024-02-05T06:27:06.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T06:26:45.309832+0000",' '"deadline":' '"2024-02-12T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:06.851 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T06:27:06.853 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-07T06:26:45.309832+0000"' 2024-02-05T06:27:06.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-07T06:26:45.309832+0000 2024-02-05T06:27:06.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-07T06:26:45.309832+0000 2024-02-05T06:27:06.853 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:06.855 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-02-05T06:27:06.856 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-07 = 2024-02-07 2024-02-05T06:27:06.857 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T06:26:45.309832+0000",' '"deadline":' '"2024-02-12T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:06.857 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T06:27:06.859 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-12T06:26:45.309832+0000"' 2024-02-05T06:27:06.859 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-12T06:26:45.309832+0000 2024-02-05T06:27:06.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-12T06:26:45.309832+0000 2024-02-05T06:27:06.860 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:06.861 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 week' 2024-02-05T06:27:06.862 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-12 = 2024-02-12 2024-02-05T06:27:06.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: get_asok_path osd.1 2024-02-05T06:27:06.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:27:06.863 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:27:06.863 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:27:06.864 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:06.864 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:06.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.1.asok 2024-02-05T06:27:06.864 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: expr 604800 '*' 2 2024-02-05T06:27:06.864 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: CEPH_ARGS= 2024-02-05T06:27:06.865 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.548953/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2024-02-05T06:27:06.944 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:27:06.944 INFO:tasks.workunit.client.0.smithi188.stdout: "success": "osd_scrub_max_interval = '' " 2024-02-05T06:27:06.944 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:27:06.951 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:193: TEST_interval_changes: sleep 15 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:194: TEST_interval_changes: check_dump_scrubs 1 '2 days' '2 week' 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=2 days' 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:27:21.955 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:27:21.956 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:21.956 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:21.956 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.1.asok 2024-02-05T06:27:21.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T06:27:21.956 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.548953/ceph-osd.1.asok dump_scrubs 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr: { 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr: "pgid": "1.0", 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr: "sched_time": "2024-02-07T06:26:45.309832+0000", 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr: "deadline": "2024-02-19T06:26:45.309832+0000", 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr: "forced": false 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr: } 2024-02-05T06:27:22.051 INFO:tasks.workunit.client.0.smithi188.stderr:]' 2024-02-05T06:27:22.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T06:26:45.309832+0000",' '"deadline":' '"2024-02-19T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:22.052 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T06:27:22.054 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-07T06:26:45.309832+0000"' 2024-02-05T06:27:22.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-07T06:26:45.309832+0000 2024-02-05T06:27:22.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-07T06:26:45.309832+0000 2024-02-05T06:27:22.054 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:22.056 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2024-02-05T06:27:22.057 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-07 = 2024-02-07 2024-02-05T06:27:22.057 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-07T06:26:45.309832+0000",' '"deadline":' '"2024-02-19T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:22.058 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T06:27:22.060 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-19T06:26:45.309832+0000"' 2024-02-05T06:27:22.060 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-19T06:26:45.309832+0000 2024-02-05T06:27:22.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-19T06:26:45.309832+0000 2024-02-05T06:27:22.061 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:22.062 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-02-05T06:27:22.063 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-19 = 2024-02-19 2024-02-05T06:27:22.063 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: expr 86400 '*' 3 2024-02-05T06:27:22.064 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2024-02-05T06:27:22.422 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 scrub_min_interval to 259200 2024-02-05T06:27:22.433 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:198: TEST_interval_changes: sleep 15 2024-02-05T06:27:37.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:199: TEST_interval_changes: check_dump_scrubs 1 '3 days' '2 week' 2024-02-05T06:27:37.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T06:27:37.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-02-05T06:27:37.435 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=2 week' 2024-02-05T06:27:37.436 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.1.asok 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T06:27:37.437 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.548953/ceph-osd.1.asok dump_scrubs 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr: { 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr: "pgid": "1.0", 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr: "sched_time": "2024-02-08T06:26:45.309832+0000", 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr: "deadline": "2024-02-19T06:26:45.309832+0000", 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr: "forced": false 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr: } 2024-02-05T06:27:37.535 INFO:tasks.workunit.client.0.smithi188.stderr:]' 2024-02-05T06:27:37.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T06:26:45.309832+0000",' '"deadline":' '"2024-02-19T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:37.536 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T06:27:37.538 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-08T06:26:45.309832+0000"' 2024-02-05T06:27:37.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-08T06:26:45.309832+0000 2024-02-05T06:27:37.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-08T06:26:45.309832+0000 2024-02-05T06:27:37.538 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:37.540 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-02-05T06:27:37.541 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-08 = 2024-02-08 2024-02-05T06:27:37.541 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T06:26:45.309832+0000",' '"deadline":' '"2024-02-19T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:37.541 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T06:27:37.544 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-19T06:26:45.309832+0000"' 2024-02-05T06:27:37.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-19T06:26:45.309832+0000 2024-02-05T06:27:37.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-19T06:26:45.309832+0000 2024-02-05T06:27:37.544 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:37.546 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 week' 2024-02-05T06:27:37.547 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-19 = 2024-02-19 2024-02-05T06:27:37.547 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: expr 604800 '*' 3 2024-02-05T06:27:37.548 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:202: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2024-02-05T06:27:37.906 INFO:tasks.workunit.client.0.smithi188.stderr:set pool 1 scrub_max_interval to 1814400 2024-02-05T06:27:37.917 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:203: TEST_interval_changes: sleep 15 2024-02-05T06:27:52.920 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:204: TEST_interval_changes: check_dump_scrubs 1 '3 days' '3 week' 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: local primary=1 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:136: check_dump_scrubs: local 'sched_time_check=3 days' 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: local 'deadline_check=3 week' 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: get_asok_path osd.1 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:52.921 INFO:tasks.workunit.client.0.smithi188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:52.922 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.1.asok 2024-02-05T06:27:52.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: CEPH_ARGS= 2024-02-05T06:27:52.922 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.548953/ceph-osd.1.asok dump_scrubs 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:139: check_dump_scrubs: DS='[ 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr: { 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr: "pgid": "1.0", 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr: "sched_time": "2024-02-08T06:26:45.309832+0000", 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr: "deadline": "2024-02-26T06:26:45.309832+0000", 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr: "forced": false 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr: } 2024-02-05T06:27:53.019 INFO:tasks.workunit.client.0.smithi188.stderr:]' 2024-02-05T06:27:53.020 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T06:26:45.309832+0000",' '"deadline":' '"2024-02-26T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:53.020 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: jq '.[0].sched_time' 2024-02-05T06:27:53.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: eval 'SCHED_TIME="2024-02-08T06:26:45.309832+0000"' 2024-02-05T06:27:53.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:141: check_dump_scrubs: SCHED_TIME=2024-02-08T06:26:45.309832+0000 2024-02-05T06:27:53.022 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: echo 2024-02-08T06:26:45.309832+0000 2024-02-05T06:27:53.023 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:53.024 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2024-02-05T06:27:53.025 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: check_dump_scrubs: test 2024-02-08 = 2024-02-08 2024-02-05T06:27:53.025 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2024-02-08T06:26:45.309832+0000",' '"deadline":' '"2024-02-26T06:26:45.309832+0000",' '"forced":' false '}' ']' 2024-02-05T06:27:53.026 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: jq '.[0].deadline' 2024-02-05T06:27:53.028 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: eval 'DEADLINE="2024-02-26T06:26:45.309832+0000"' 2024-02-05T06:27:53.028 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: check_dump_scrubs: DEADLINE=2024-02-26T06:26:45.309832+0000 2024-02-05T06:27:53.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: echo 2024-02-26T06:26:45.309832+0000 2024-02-05T06:27:53.029 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2024-02-05T06:27:53.030 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 week' 2024-02-05T06:27:53.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:145: check_dump_scrubs: test 2024-02-26 = 2024-02-26 2024-02-05T06:27:53.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:205: TEST_interval_changes: perf_counters td/osd-scrub-test 2 2024-02-05T06:27:53.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2024-02-05T06:27:53.031 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=2 2024-02-05T06:27:53.032 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 2 - 1 2024-02-05T06:27:53.032 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 1 2024-02-05T06:27:53.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:27:53.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2024-02-05T06:27:53.034 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:27:53.127 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:27:53.127 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:27:53.127 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.127 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.127 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.128 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.129 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.130 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.131 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.132 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.133 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:27:53.134 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.135 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.136 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.137 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.138 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.139 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.140 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.141 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.142 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.142 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:27:53.142 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:27:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-02-05T06:27:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2024-02-05T06:27:53.142 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2024-02-05T06:27:53.229 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:27:53.229 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_ec": [ 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.230 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.231 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.232 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_dp_repl": [ 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "deep", 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.233 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.234 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.235 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_ec": [ 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "ec" 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.236 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.237 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.238 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: ], 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "osd_scrub_sh_repl": [ 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: { 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "labels": { 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "level": "shallow", 2024-02-05T06:27:53.239 INFO:tasks.workunit.client.0.smithi188.stdout: "pooltype": "replicated" 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "counters": { 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_started": 0, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "num_scrubs_past_reservation": 0, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs": 0, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_scrubs_elapsed": { 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs": 0, 2024-02-05T06:27:53.240 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_scrubs_elapsed": { 2024-02-05T06:27:53.241 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.241 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.241 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.241 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.241 INFO:tasks.workunit.client.0.smithi188.stdout: "preemptions": 0, 2024-02-05T06:27:53.241 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_selected": 0, 2024-02-05T06:27:53.241 INFO:tasks.workunit.client.0.smithi188.stdout: "chunk_busy": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "locked_object": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "write_blocked_by_scrub": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "scrub_reservations_completed": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "successful_reservations_elapsed": { 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_aborted": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_timed_out": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_failure": 0, 2024-02-05T06:27:53.242 INFO:tasks.workunit.client.0.smithi188.stdout: "reservation_process_skipped": 0, 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: "failed_reservations_elapsed": { 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: "avgcount": 0, 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: "sum": 0E-9, 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: "avgtime": 0E-9 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: }, 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: "replicas_in_reservation": 0 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: } 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout: ] 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:27:53.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_interval_changes ------------------' 2024-02-05T06:27:53.244 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Teardown Test TEST_interval_changes ------------------ 2024-02-05T06:27:53.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2024-02-05T06:27:53.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:27:53.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:27:53.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:27:53.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:27:53.245 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:27:53.245 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:27:53.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:27:53.246 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:27:53.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:27:53.347 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:27:53.347 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:27:53.348 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:27:53.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:27:53.349 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:27:53.349 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:27:53.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:53.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:27:53.350 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:27:53.350 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:53.351 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:27:53.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:27:53.352 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:27:53.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:27:53.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:53.362 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:53.362 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:27:53.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:27:53.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:27:53.363 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_interval_changes ------------------' 2024-02-05T06:27:53.364 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Complete Test TEST_interval_changes ------------------ 2024-02-05T06:27:53.364 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2024-02-05T06:27:53.364 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Prepare Test TEST_just_deep_scrubs ------------------- 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_just_deep_scrubs -------------------' 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-test 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-test 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:27:53.366 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:27:53.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:27:53.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:27:53.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:27:53.367 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:27:53.367 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:27:53.368 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:27:53.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:27:53.368 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:27:53.368 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:27:53.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:53.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:27:53.369 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:27:53.369 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:27:53.370 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:27:53.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-02-05T06:27:53.371 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:27:53.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:27:53.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:53.372 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:53.372 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:27:53.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:27:53.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:27:53.373 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-test 2024-02-05T06:27:53.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-02-05T06:27:53.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:27:53.374 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:27:53.374 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.548953 2024-02-05T06:27:53.375 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-02-05T06:27:53.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-02-05T06:27:53.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-02-05T06:27:53.376 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2024-02-05T06:27:53.376 INFO:tasks.workunit.client.0.smithi188.stdout:-------------- Run Test TEST_just_deep_scrubs ----------------------- 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_just_deep_scrubs -----------------------' 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_just_deep_scrubs td/osd-scrub-test 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:467: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:468: TEST_just_deep_scrubs: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:468: TEST_just_deep_scrubs: local -A cluster_conf 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-test 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-02-05T06:27:53.378 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-02-05T06:27:53.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-02-05T06:27:53.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-02-05T06:27:53.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-02-05T06:27:53.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-02-05T06:27:53.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-02-05T06:27:53.379 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-02-05T06:27:54.921 INFO:tasks.workunit.client.0.smithi188.stdout:add osd0 e8dd77f4-e781-4637-8b0e-da657adc5895 2024-02-05T06:27:55.138 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:27:55.178 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:27:55.182+0000 7feb414c54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:55.189 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:27:55.193+0000 7feb414c54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:55.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:27:55.195+0000 7feb414c54c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:27:55.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:27:55.195+0000 7feb414c54c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:27:55.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:27:55.195+0000 7feb414c54c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:27:55.190 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:27:55.195+0000 7feb414c54c0 -1 bluestore(td/osd-scrub-test/0/block) _read_bdev_label failed to open td/osd-scrub-test/0/block: (2) No such file or directory 2024-02-05T06:27:55.272 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:27:55.277+0000 7feb414c54c0 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2024-02-05T06:28:00.821 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd0 key to auth repository 2024-02-05T06:28:01.168 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.0 2024-02-05T06:28:01.196 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:01.201+0000 7f34549074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:01.198 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:01.203+0000 7f34549074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:01.199 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:01.204+0000 7f34549074c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:01.435 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:28:01.729 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:01.734+0000 7f34549074c0 -1 Falling back to public interface 2024-02-05T06:28:02.703 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:28:03.972 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:28:04.105 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:04.110+0000 7f34549074c0 -1 osd.0 0 log_to_monitors true 2024-02-05T06:28:05.296 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:28:06.574 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:28:07.852 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:28:09.142 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:28:10.420 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:28:11.699 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:28:12.973 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:28:14.247 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:28:15.522 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:28:16.973 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:28:18.633 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:28:20.403 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:28:21.940 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:28:23.306 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:28:24.732 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:28:26.433 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:28:27.824 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:28:28.088 INFO:tasks.workunit.client.0.smithi188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3981319044,v1:127.0.0.1:6803/3981319044] [v2:127.0.0.1:6804/3981319044,v1:127.0.0.1:6805/3981319044] exists,up e8dd77f4-e781-4637-8b0e-da657adc5895 2024-02-05T06:28:28.092 INFO:tasks.workunit.client.0.smithi188.stdout:add osd1 4da63e0a-6c1e-4123-9260-f092175ceb13 2024-02-05T06:28:28.465 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:28:28.506 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:28.510+0000 7fc27b8114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:28.541 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:28.545+0000 7fc27b8114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:28.542 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:28.546+0000 7fc27b8114c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:28.543 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:28.547+0000 7fc27b8114c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:28:28.543 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:28.547+0000 7fc27b8114c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:28:28.543 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:28.547+0000 7fc27b8114c0 -1 bluestore(td/osd-scrub-test/1/block) _read_bdev_label failed to open td/osd-scrub-test/1/block: (2) No such file or directory 2024-02-05T06:28:28.600 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:28.604+0000 7fc27b8114c0 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2024-02-05T06:28:34.589 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd1 key to auth repository 2024-02-05T06:28:34.933 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.1 2024-02-05T06:28:34.960 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:34.964+0000 7fb0928254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:34.962 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:34.966+0000 7fb0928254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:34.963 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:34.967+0000 7fb0928254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:28:35.198 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:28:35.486 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:35.490+0000 7fb0928254c0 -1 Falling back to public interface 2024-02-05T06:28:36.461 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:28:37.730 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:28:37.830 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:28:37.834+0000 7fb0928254c0 -1 osd.1 0 log_to_monitors true 2024-02-05T06:28:39.057 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:28:40.330 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:28:41.677 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:28:42.962 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:28:44.336 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:28:45.660 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:28:46.938 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:28:48.271 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:28:49.643 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:28:50.922 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:28:52.281 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:28:53.784 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:28:55.376 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:28:56.969 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:28:58.339 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:29:00.068 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:29:01.789 INFO:tasks.workunit.client.0.smithi188.stdout:19 2024-02-05T06:29:02.073 INFO:tasks.workunit.client.0.smithi188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3574533912,v1:127.0.0.1:6811/3574533912] [v2:127.0.0.1:6812/3574533912,v1:127.0.0.1:6813/3574533912] exists,up 4da63e0a-6c1e-4123-9260-f092175ceb13 2024-02-05T06:29:02.077 INFO:tasks.workunit.client.0.smithi188.stdout:add osd2 a56c9f40-c944-4d16-904c-92474955e48e 2024-02-05T06:29:02.474 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:29:02.514 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:02.518+0000 7f42870254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:29:02.575 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:02.579+0000 7f42870254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:29:02.576 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:02.580+0000 7f42870254c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:29:02.577 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:02.580+0000 7f42870254c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:29:02.577 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:02.580+0000 7f42870254c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:29:02.577 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:02.580+0000 7f42870254c0 -1 bluestore(td/osd-scrub-test/2/block) _read_bdev_label failed to open td/osd-scrub-test/2/block: (2) No such file or directory 2024-02-05T06:29:02.645 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:02.649+0000 7f42870254c0 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2024-02-05T06:29:08.609 INFO:tasks.workunit.client.0.smithi188.stdout:adding osd2 key to auth repository 2024-02-05T06:29:08.954 INFO:tasks.workunit.client.0.smithi188.stdout:start osd.2 2024-02-05T06:29:08.982 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:08.985+0000 7f9fd7c9e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:29:08.984 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:08.987+0000 7f9fd7c9e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:29:08.985 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:08.988+0000 7f9fd7c9e4c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-02-05T06:29:09.233 INFO:tasks.workunit.client.0.smithi188.stdout:0 2024-02-05T06:29:09.503 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:09.507+0000 7f9fd7c9e4c0 -1 Falling back to public interface 2024-02-05T06:29:10.498 INFO:tasks.workunit.client.0.smithi188.stdout:1 2024-02-05T06:29:11.770 INFO:tasks.workunit.client.0.smithi188.stdout:2 2024-02-05T06:29:11.834 INFO:tasks.workunit.client.0.smithi188.stderr:2024-02-05T06:29:11.838+0000 7f9fd7c9e4c0 -1 osd.2 0 log_to_monitors true 2024-02-05T06:29:13.051 INFO:tasks.workunit.client.0.smithi188.stdout:3 2024-02-05T06:29:14.470 INFO:tasks.workunit.client.0.smithi188.stdout:4 2024-02-05T06:29:15.794 INFO:tasks.workunit.client.0.smithi188.stdout:5 2024-02-05T06:29:17.068 INFO:tasks.workunit.client.0.smithi188.stdout:6 2024-02-05T06:29:18.454 INFO:tasks.workunit.client.0.smithi188.stdout:7 2024-02-05T06:29:19.842 INFO:tasks.workunit.client.0.smithi188.stdout:8 2024-02-05T06:29:21.119 INFO:tasks.workunit.client.0.smithi188.stdout:9 2024-02-05T06:29:22.478 INFO:tasks.workunit.client.0.smithi188.stdout:10 2024-02-05T06:29:23.850 INFO:tasks.workunit.client.0.smithi188.stdout:11 2024-02-05T06:29:25.131 INFO:tasks.workunit.client.0.smithi188.stdout:12 2024-02-05T06:29:27.012 INFO:tasks.workunit.client.0.smithi188.stdout:13 2024-02-05T06:29:28.851 INFO:tasks.workunit.client.0.smithi188.stdout:14 2024-02-05T06:29:30.331 INFO:tasks.workunit.client.0.smithi188.stdout:15 2024-02-05T06:29:32.049 INFO:tasks.workunit.client.0.smithi188.stdout:16 2024-02-05T06:29:33.727 INFO:tasks.workunit.client.0.smithi188.stdout:17 2024-02-05T06:29:35.010 INFO:tasks.workunit.client.0.smithi188.stdout:18 2024-02-05T06:29:35.312 INFO:tasks.workunit.client.0.smithi188.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2183817479,v1:127.0.0.1:6819/2183817479] [v2:127.0.0.1:6820/2183817479,v1:127.0.0.1:6821/2183817479] exists,up a56c9f40-c944-4d16-904c-92474955e48e 2024-02-05T06:29:35.704 INFO:tasks.workunit.client.0.smithi188.stderr:pool 'test' created 2024-02-05T06:29:37.583 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.0 seq 21474836496 2024-02-05T06:29:39.177 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.1 seq 42949672976 2024-02-05T06:29:39.527 INFO:tasks.workunit.client.0.smithi188.stdout:waiting osd.2 seq 64424509443 2024-02-05T06:29:42.673 INFO:tasks.workunit.client.0.smithi188.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-02-05T06:29:42.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:475: TEST_just_deep_scrubs: local poolid=1 2024-02-05T06:29:42.674 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: local poolname=test 2024-02-05T06:29:42.674 INFO:tasks.workunit.client.0.smithi188.stdout:Pool: test : 1 2024-02-05T06:29:42.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2024-02-05T06:29:42.675 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: TESTDATA=testdata.548953 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:480: TEST_just_deep_scrubs: local objects=15 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.548953 bs=1032 count=1 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records in 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr:1+0 records out 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 4.4527e-05 s, 23.2 MB/s 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: seq 1 15 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:42.676 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj1 testdata.548953 2024-02-05T06:29:42.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:42.831 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj2 testdata.548953 2024-02-05T06:29:43.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:43.016 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj3 testdata.548953 2024-02-05T06:29:43.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:43.175 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj4 testdata.548953 2024-02-05T06:29:43.317 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:43.318 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj5 testdata.548953 2024-02-05T06:29:43.451 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:43.452 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj6 testdata.548953 2024-02-05T06:29:43.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:43.577 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj7 testdata.548953 2024-02-05T06:29:43.703 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:43.704 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj8 testdata.548953 2024-02-05T06:29:43.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:43.871 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj9 testdata.548953 2024-02-05T06:29:44.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:44.022 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj10 testdata.548953 2024-02-05T06:29:44.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:44.156 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj11 testdata.548953 2024-02-05T06:29:44.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:44.290 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj12 testdata.548953 2024-02-05T06:29:44.424 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:44.425 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj13 testdata.548953 2024-02-05T06:29:44.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:44.550 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj14 testdata.548953 2024-02-05T06:29:44.692 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:482: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2024-02-05T06:29:44.693 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:484: TEST_just_deep_scrubs: rados -p test put obj15 testdata.548953 2024-02-05T06:29:44.834 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: rm -f testdata.548953 2024-02-05T06:29:44.835 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:491: TEST_just_deep_scrubs: ceph osd set noscrub 2024-02-05T06:29:45.335 INFO:tasks.workunit.client.0.smithi188.stderr:noscrub is set 2024-02-05T06:29:45.346 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:492: TEST_just_deep_scrubs: ceph osd set nodeep-scrub 2024-02-05T06:29:45.712 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is set 2024-02-05T06:29:45.723 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:493: TEST_just_deep_scrubs: sleep 6 2024-02-05T06:29:51.725 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:494: TEST_just_deep_scrubs: date -Ins 2024-02-05T06:29:51.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:494: TEST_just_deep_scrubs: local now_is=2024-02-05T06:29:51,729736487+00:00 2024-02-05T06:29:51.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:495: TEST_just_deep_scrubs: declare -A sched_data 2024-02-05T06:29:51.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:496: TEST_just_deep_scrubs: local pgid=1.2 2024-02-05T06:29:51.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:499: TEST_just_deep_scrubs: set_query_debug 1.2 2024-02-05T06:29:51.726 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:294: set_query_debug: local pgid=1.2 2024-02-05T06:29:51.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: ceph pg dump pgs_brief 2024-02-05T06:29:51.727 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-02-05T06:29:51.986 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs_brief 2024-02-05T06:29:51.998 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:296: set_query_debug: local prim_osd=0 2024-02-05T06:29:51.998 INFO:tasks.workunit.client.0.smithi188.stdout:Setting scrub debug data. Primary for 1.2 is 0 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:298: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: get_asok_path osd.0 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.548953/ceph-osd.0.asok 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: CEPH_ARGS= 2024-02-05T06:29:52.000 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.548953/ceph-osd.0.asok scrubdebug 1.2 set sessions 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:501: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-02-05T06:29:51,729736487+00:00 2024-02-05T06:29:51,729736487+00:00 sched_data 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-02-05T06:29:51,729736487+00:00 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-02-05T06:29:51,729736487+00:00 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-02-05T06:29:52.088 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-02-05T06:29:52.334 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:{"success":true}( 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=23 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=18 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:29:35.658' 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:29:35.658566+0000' 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:29:52.452 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699072 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:29:35.658566+0000' 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=21 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=16 2024-02-05T06:29:52.453 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:29:52.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:502: TEST_just_deep_scrubs: local saved_last_stamp=2024-02-05T06:29:35.658566+0000 2024-02-05T06:29:52.455 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:503: TEST_just_deep_scrubs: local dbg_counter_at_start=5699072 2024-02-05T06:29:52.456 INFO:tasks.workunit.client.0.smithi188.stdout:test counter @ start: 5699072 2024-02-05T06:29:52.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:504: TEST_just_deep_scrubs: echo 'test counter @ start: 5699072' 2024-02-05T06:29:52.456 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: ceph tell 1.2 schedule-deep-scrub 2024-02-05T06:29:52.549 INFO:tasks.workunit.client.0.smithi188.stdout:{ 2024-02-05T06:29:52.549 INFO:tasks.workunit.client.0.smithi188.stdout: "deep": true, 2024-02-05T06:29:52.549 INFO:tasks.workunit.client.0.smithi188.stdout: "must": false, 2024-02-05T06:29:52.550 INFO:tasks.workunit.client.0.smithi188.stdout: "stamp": "2024-01-22T06:28:12.552880+0000" 2024-02-05T06:29:52.550 INFO:tasks.workunit.client.0.smithi188.stdout:} 2024-02-05T06:29:52.558 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:508: TEST_just_deep_scrubs: sleep 5 2024-02-05T06:29:57.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:509: TEST_just_deep_scrubs: declare -A sc_data_2 2024-02-05T06:29:57.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_just_deep_scrubs: extract_published_sch 1.2 2024-02-05T06:29:51,729736487+00:00 2024-02-05T06:29:51,729736487+00:00 sc_data_2 2024-02-05T06:29:57.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2024-02-05T06:29:57.560 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2024-02-05T06:29:57.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2024-02-05T06:29:51,729736487+00:00 2024-02-05T06:29:57.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2024-02-05T06:29:51,729736487+00:00 2024-02-05T06:29:57.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=1 2024-02-05T06:29:57.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2024-02-05T06:29:57.561 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2024-02-05T06:29:57.818 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=23 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=22 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='queued for deep scrub' 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='0' 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-01-22T06:28:12.552880+0000' 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=false 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=false 2024-02-05T06:29:57.936 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699072 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='queued for deep scrub' 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='0' 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=false 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=false 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=23 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=21 2024-02-05T06:29:57.937 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:29:57.939 INFO:tasks.workunit.client.0.smithi188.stdout:test counter @ should show no change: 5699072 2024-02-05T06:29:57.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:511: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 5699072 2024-02-05T06:29:57.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_just_deep_scrubs: (( 0 == 0 )) 2024-02-05T06:29:57.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_just_deep_scrubs: (( 5699072 == 5699072 )) 2024-02-05T06:29:57.939 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:516: TEST_just_deep_scrubs: ceph osd unset nodeep-scrub 2024-02-05T06:29:58.311 INFO:tasks.workunit.client.0.smithi188.stderr:nodeep-scrub is unset 2024-02-05T06:29:58.322 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:517: TEST_just_deep_scrubs: sleep 5 2024-02-05T06:30:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2024-02-05T06:30:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: declare -A expct_qry_duration 2024-02-05T06:30:03.324 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:519: TEST_just_deep_scrubs: sc_data_2=() 2024-02-05T06:30:03.325 INFO:tasks.workunit.client.0.smithi188.stdout:test counter @ should be higher than before the unset: 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2024-02-05T06:29:35.658566+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:98: wait_any_cond: local pgid=1.2 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:99: wait_any_cond: local retries=10 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:100: wait_any_cond: local cmp_date=2024-02-05T06:29:35.658566+0000 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:101: wait_any_cond: local -n ep=expct_qry_duration 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:102: wait_any_cond: local -n out_array=sc_data_2 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:103: wait_any_cond: local -A sc_data 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:104: wait_any_cond: local extr_dbg=2 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local saved_echo_flag=x 2024-02-05T06:30:03.326 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: set +x 2024-02-05T06:30:03.327 INFO:tasks.workunit.client.0.smithi188.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2024-02-05T06:29:35.658566+0000 (10 retries) 2024-02-05T06:30:04.086 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:04.204 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:04.205 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:04.207 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 1 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:04.207 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:04.963 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:05.108 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:05.108 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:05.108 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:05.108 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:05.109 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:05.110 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:05.110 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:05.110 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:05.112 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 2 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:05.112 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:05.864 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:05.984 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:05.985 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:05.986 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 3 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:05.986 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:06.738 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:06.856 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:06.857 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:06.859 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 4 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:06.859 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:07.613 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:07.732 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:07.733 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:07.735 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 5 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:07.735 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:08.488 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:08.606 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:08.607 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:08.609 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 6 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:08.610 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:09.360 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:09.478 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:09.478 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:09.479 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:09.480 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:09.480 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:09.480 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:09.480 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:09.480 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:09.480 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:09.482 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 7 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:09.482 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:10.254 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:10.371 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:10.372 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:10.373 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:10.373 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:10.373 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:10.373 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:10.373 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:10.373 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:10.375 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 8 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:10.375 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:11.127 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:11.245 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:11.245 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:11.245 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:11.245 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:11.245 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:11.246 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:11.247 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:11.247 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:11.249 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 9 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:11.249 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:12.002 INFO:tasks.workunit.client.0.smithi188.stderr:dumped pgs 2024-02-05T06:30:12.123 INFO:tasks.workunit.client.0.smithi188.stdout:( 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_epoch]=24 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_seq]=31 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_active]=false 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule]='scrub scheduled' 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_schedule_at]='2024-02-06T06:30:00.324' 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_duration]=0 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_stamp]='2024-02-05T06:30:00.324714+0000' 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_last_scrub]='0x0' 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_is_future]=true 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_vs_date]=true 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[query_scrub_seq]=5699075 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_pg_state]='active+clean' 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_state_has_scrubbing]=false 2024-02-05T06:30:12.124 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_last_duration]=0 2024-02-05T06:30:12.125 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule]='periodic scrub scheduled' 2024-02-05T06:30:12.125 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_schedule_at]='2024-02-06T06:30:00.324714+0000' 2024-02-05T06:30:12.125 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_is_future]=true 2024-02-05T06:30:12.125 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_vs_date]=true 2024-02-05T06:30:12.125 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_reported_epoch]=24 2024-02-05T06:30:12.125 INFO:tasks.workunit.client.0.smithi188.stdout:[dmp_seq]=30 2024-02-05T06:30:12.125 INFO:tasks.workunit.client.0.smithi188.stdout:) 2024-02-05T06:30:12.127 INFO:tasks.workunit.client.0.smithi188.stdout:--> loop: 10 ~ false / 31 / 30 / true / 2024-02-05T06:30:00.324714+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2024-02-05T06:30:12.127 INFO:tasks.workunit.client.0.smithi188.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 0 2024-02-05T06:30:12.127 INFO:tasks.workunit.client.0.smithi188.stdout:WaitingAfterScrub : wait_any_cond(): failure. Note: query-active=false 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:156: wait_any_cond: return 1 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: return 1 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: return 1 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-test 1 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-test 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-test KILL 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-02-05T06:30:12.129 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-02-05T06:30:12.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-02-05T06:30:12.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-02-05T06:30:12.130 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-02-05T06:30:12.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-02-05T06:30:12.236 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-02-05T06:30:12.236 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-02-05T06:30:12.237 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-02-05T06:30:12.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-02-05T06:30:12.238 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-02-05T06:30:12.238 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-02-05T06:30:12.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:30:12.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-02-05T06:30:12.239 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-02-05T06:30:12.239 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:30:12.240 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:30:12.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-02-05T06:30:12.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-02-05T06:30:12.241 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-02-05T06:30:12.242 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-test/mgr.x.log td/osd-scrub-test/mon.a.log td/osd-scrub-test/osd.0.log td/osd-scrub-test/osd.1.log td/osd-scrub-test/osd.2.log /home/ubuntu/cephtest/archive/log 2024-02-05T06:30:12.243 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-test 2024-02-05T06:30:12.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-02-05T06:30:12.251 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-02-05T06:30:12.252 INFO:tasks.workunit.client.0.smithi188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.548953 2024-02-05T06:30:12.252 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.548953 2024-02-05T06:30:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:30:12.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-02-05T06:30:12.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-02-05T06:30:12.253 INFO:tasks.workunit.client.0.smithi188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-02-05T06:30:12.254 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-02-05T06:30:12.254 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-05T06:30:12.515 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-02-05T06:30:17.335 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c9d98677c75140bc92cb0a3e021baf6d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3cfd0d29cc3905ea860fc5482a5a58407e15a0d7/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' 2024-02-05T06:30:17.338 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-05T06:30:17.378 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-05T06:30:17.378 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T06:30:17.438 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-05T06:30:17.446 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-05T06:30:17.446 DEBUG:teuthology.orchestra.run.smithi188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T06:30:17.467 INFO:teuthology.orchestra.run.smithi188.stderr:bash: line 1: ntpq: command not found 2024-02-05T06:30:17.472 INFO:teuthology.orchestra.run.smithi188.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T06:30:17.472 INFO:teuthology.orchestra.run.smithi188.stdout:=============================================================================== 2024-02-05T06:30:17.472 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 +866us[ +866us] +/- 91ms 2024-02-05T06:30:17.473 INFO:teuthology.orchestra.run.smithi188.stdout:^* hv02.front.sepia.ceph.com 3 7 377 120 -1637us[-1645us] +/- 75ms 2024-02-05T06:30:17.473 INFO:teuthology.orchestra.run.smithi188.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +1333us[+1333us] +/- 68ms 2024-02-05T06:30:17.473 INFO:teuthology.orchestra.run.smithi188.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:30:17.473 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-05T06:30:17.481 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-05T06:30:17.482 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-05T06:30:17.503 DEBUG:teuthology.orchestra.run.smithi188:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T06:30:17.577 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T06:30:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:30:17.607 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-05T06:30:17.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-05T06:30:17.637 INFO:teuthology.task.internal:Duration was 7547.452836 seconds 2024-02-05T06:30:17.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-05T06:30:17.672 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-05T06:30:17.672 DEBUG:teuthology.orchestra.run.smithi188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T06:30:17.716 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T06:30:18.191 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-05T06:30:18.192 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi188.front.sepia.ceph.com 2024-02-05T06:30:18.192 DEBUG:teuthology.orchestra.run.smithi188:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T06:30:18.252 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-05T06:30:18.253 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T06:30:18.322 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-05T06:30:18.322 DEBUG:teuthology.orchestra.run.smithi188:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T06:30:18.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-05T06:30:18.469 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-05T06:30:18.470 DEBUG:teuthology.orchestra.run.smithi188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T06:30:18.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-05T06:30:18.505 DEBUG:teuthology.orchestra.run.smithi188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:30:18.562 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = core 2024-02-05T06:30:18.582 DEBUG:teuthology.orchestra.run.smithi188:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T06:30:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:30:18.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-05T06:30:18.645 INFO:teuthology.task.internal:Transferring archived files... 2024-02-05T06:30:18.647 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404/remote/smithi188 2024-02-05T06:30:18.648 DEBUG:teuthology.orchestra.run.smithi188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T06:30:19.449 INFO:teuthology.task.internal:Removing archive directory... 2024-02-05T06:30:19.450 DEBUG:teuthology.orchestra.run.smithi188:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T06:30:19.488 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-05T06:30:19.498 INFO:teuthology.task.internal:Not uploading archives. 2024-02-05T06:30:19.498 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-05T06:30:19.539 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-05T06:30:19.539 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T06:30:19.554 INFO:teuthology.orchestra.run.smithi188.stdout: 265294 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 06:30 /home/ubuntu/cephtest 2024-02-05T06:30:19.554 INFO:teuthology.orchestra.run.smithi188.stdout: 400573 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 04:31 /home/ubuntu/cephtest/mnt.0 2024-02-05T06:30:19.554 INFO:teuthology.orchestra.run.smithi188.stdout: 400574 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 06:15 /home/ubuntu/cephtest/mnt.0/client.0 2024-02-05T06:30:19.554 INFO:teuthology.orchestra.run.smithi188.stdout: 400616 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 5 06:29 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-02-05T06:30:19.554 INFO:teuthology.orchestra.run.smithi188.stdout: 412448 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 06:30 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-02-05T06:30:19.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:30:19.555 INFO:teuthology.orchestra.run.smithi188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-05T06:30:19.555 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi188 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-05T06:30:19.606 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-05T06:30:19.617 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-05T06:30:19.639 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi188 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-02-05T06:30:19.640 INFO:teuthology.nuke:Checking targets against current locks 2024-02-05T06:30:19.666 DEBUG:teuthology.nuke:shortname: smithi188 2024-02-05T06:30:19.666 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T06:30:19.690 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi188.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdiazbou-2024-01-29_15:40:07-rados-wip-pereman2-buffer-space-distro-default-smithi/7536404', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 04:15:29.696964', 'locked_by': 'scheduled_pdiazbou@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDorhZVwQxTa4xYl3O/TUIrw9WBMlK2qSvz5WHTkS8iDHdf2Iips913ViLWYawfBhXCurZRqRwd8qNWadnAlVaM='} 2024-02-05T06:30:19.734 INFO:teuthology.orchestra.console.smithi188:Power off 2024-02-05T06:30:19.734 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T06:30:19.756 DEBUG:teuthology.orchestra.console.smithi188:power off output: Chassis Power Control: Down/Off 2024-02-05T06:30:19.757 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T06:30:19.779 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-02-05T06:30:23.781 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T06:30:23.891 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is on 2024-02-05T06:30:27.894 DEBUG:teuthology.orchestra.console.smithi188:pexpect command: ipmitool -H smithi188.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T06:30:28.004 DEBUG:teuthology.orchestra.console.smithi188:check power output: Chassis Power is off 2024-02-05T06:30:28.105 INFO:teuthology.orchestra.console.smithi188:Power off completed 2024-02-05T06:30:28.210 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 7547.452835798264 failure_reason: 'Command failed (workunit test scrub/osd-scrub-test.sh) on smithi188 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3cfd0d29cc3905ea860fc5482a5a58407e15a0d7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh''' flavor: default owner: scheduled_pdiazbou@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c9d98677c75140bc92cb0a3e021baf6d status: fail success: false 2024-02-05T06:30:28.211 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T06:30:28.295 INFO:teuthology.run:FAIL